Debugging Local LLM Benchmarks: Aider, llama.cpp, and Runaway Output

A local LLM benchmark measures more than model weights. It also measures the quantized artifact, chat template, reasoning parser, API adapter, sampling parameters, context limits, evaluator, and agent loop around the model.

I ran into each of those variables while testing nine local models for tool use, structured output, debugging, and code editing. Reasoning models initially returned no gradable final answer. Exact-match validators rejected outputs that were semantically usable. Some Aider requests generated tens of thousands of tokens before stopping. Changing the sampling configuration improved the observed behavior, but it did not by itself prove a universal cause.

This article documents the evaluation design and the failures that had to be classified before the scores became interpretable. It does not declare an overall model winner. The completed Aider Polyglot comparison belongs in Part 2.

Test System And Artifacts

The final Aider run started on this desktop configuration:

The roster is heterogeneous: total parameter counts, active parameter counts, architectures, templates, and quantizations differ. The comparison is therefore between the exact local artifacts and serving profiles below, not abstract model families.

An endpoint ID is not enough to reproduce a result. The GGUF filename, quantization, template, template arguments, multimodal projector, and reasoning mode must travel with the score.

Three Evaluation Layers

No single suite covered the operational questions I cared about, so I used three complementary layers. The diagram is conceptual; the layers are not a formal statistical pipeline.

Layer 1: BenchLocal Screening

I selected seven 15-scenario packs:

• ToolCall-15 v1.0.1: tool selection, arguments, chains, restraint, and recovery;
• StructOutput-15 v1.0.0: machine-readable output contracts;
• InstructFollow-15 v1.0.0: multi-constraint instruction following;
• PromptAuthority-15 v1.0.0: lower-authority prompt conflicts;
• BugFind-15 v1.0.1: execution-backed debugging;
• DataExtract-15 v1.0.0: strict data extraction;
• ReasonMath-15 v1.0.0: applied arithmetic and reasoning.

That produced 105 checks per model. The packs measure different capabilities, so averaging them does not create a statistically meaningful general score.

Layer 2: A Fixed Single-Call API Harness

The local Python harness calls /v1/chat/completions or /v1/responses, stores JSONL results and raw responses, and applies weighted validators. Its 12 tasks cover exact response formats, bug identification, test interpretation, noisy context retrieval, memory scope, research synthesis, safe tool planning, and unified diffs.

Patch evaluation has three separate stages:

1. inspect the requested diff structure;
2. run git apply --check and apply the patch in an isolated copy;
3. execute a behavioral assertion against the modified files.

This distinction matters because a valid patch can violate a requested textual convention, while a plausible-looking diff can fail to apply or preserve the bug.

Layer 3: Aider Polyglot

The final layer uses the public Aider Polyglot exercises in diff edit mode. The local dataset contains 225 exercises across C++, Go, Java, JavaScript, Python, and Rust. Aider supplies repository context to the model and applies the proposed edit; the benchmark then evaluates the resulting code with exercise tests.

This is the most expensive layer and the closest of the three to the coding workflow being evaluated. Its completed results will be reported separately.

Reasoning Output Is Part Of The Protocol

AgentWorld and Ornith can emit a reasoning segment before a final answer. In an early Responses API run with an 800-token output budget, the server returned a reasoning item but no assistant message. The grader extracted no final text and correctly scored the contract as failed. That run did not isolate whether the model would have completed with a larger budget.

The later serving profile enabled and preserved thinking for these two models. In the July 23 raw responses, llama.cpp exposed a completed reasoning item and a separate assistant message containing output_text. The ordinary Qwen3.6 and Gemma profiles explicitly disabled thinking; other models used their configured templates.

Those are different operating profiles, not controlled architecture-only conditions. Any comparison must record:

• template and template arguments;
• whether thinking is enabled and preserved;
• how reasoning and final text are separated;
• output budget and final-message presence;
• which portion of the response reaches the validator.

What The Screening Runs Showed

BenchLocal Capability Profile

The following matrix is useful for finding strengths and failure modes, not for ranking the models. Seven models come from the original complete runs; AgentWorld and Ornith come from later reasoning-enabled runs. Qwen3-Coder-Next also changed quantization before the final Aider run. Pairwise differences in this table can therefore include serving and artifact changes.

The profile exposes trade-offs that an average would hide. Qwen3-Coder-Next had the highest BugFind score but weak instruction and authority scores. Agents-A1 combined a perfect ToolCall result with a DataExtract score of 34; one recurring failure was wrapping otherwise usable JSON in a Markdown fence. No model scored above 67 on PromptAuthority.

PromptAuthority is only a 15-case screening pack, not a security evaluation. Tool allowlists, approval gates, sandboxing, and untrusted-content isolation are required regardless of the model score.

Custom Harness Snapshot

On July 23, I ran each model once on all 12 custom tasks:

uv run python -m local_model_bench.run --api responses --max-tokens 32768

The run used the Responses API, temperature=0, and a 32,768-token output cap. No API errors were recorded. The table reports hard passes, the mean weighted validator score, and observed end-to-end request latency. Latency includes each profile’s reasoning output and any loading overhead visible to the client; it is not a normalized throughput measurement.

This is a diagnostic result from 12 hand-authored tasks and one sample per model. It has no confidence intervals and does not justify a general model ranking. Its strict validators also expose contract failures that are not always semantic failures. Every model returned API: Responses API where one task required the exact lowercase string API: responses. Qwen3.6 Unsloth returned an applicable and behaviorally correct patch but omitted the requested a/ and b/ filename prefixes, so it received a partial rather than hard pass.

For this reason, the harness retains both check-level scores and binary thresholds. Flattening the results to pass/fail would discard useful evidence; combining them into an arbitrary composite ranking would add false precision.

Runaway Output During Aider Runs

The most expensive failure mode appeared in early Aider runs. Some responses produced plausible code or prose and then repeated a block until reaching the 32,768-token output cap or the 1,800-second request timeout. With llama.cpp set to parallel=1, one such request occupied the only decoding slot.

In one earlier Qwen3-Coder-Next run, I observed a 92-line block repeated 32 times, consuming roughly 30,000 output tokens. The raw transcript was not retained. It is therefore an operational observation, not a reproducible test case. Separately, later server logs provide direct evidence of exact token-tail repetition detected by the patched runtime.

Changing timeout or output limits only changes the cost of this failure:

• a longer timeout permits more generation;
• a larger output cap permits more tokens;
• a smaller cap can truncate legitimate large edits.

These controls are still useful containment, but they do not identify why a generation entered a repetitive trajectory.

What Was Proven About Greedy Decoding

Inspection of the effective Aider parameters showed temperature=0 before the local model settings overrode it. The same benchmark container later reported:

{'max_tokens': 32768, 'temperature': 0.7}

The llama.cpp source used during the investigation treats numeric zero as a present value, not as an omitted server default. At temperature <= 0, the temperature sampler retains the maximum-logit candidate and sets the others to negative infinity. In this stack, temperature=0 therefore meant greedy decoding.

The benchmark override was:

- name: aider/extra_params
  extra_params:
    max_tokens: 32768
    temperature: 0.7

Later runs at 0.7 completed faster and did not reproduce the same repeated blocks in the cases I observed. That is useful operational evidence, but it is not a controlled causal result: the retained records do not hold every prompt, seed, artifact, and runtime condition constant. The defensible conclusion is narrower:

1. Aider sent zero in the earlier effective configuration.
2. llama.cpp interpreted zero as greedy decoding.
3. repetitive output was observed under that configuration.
4. the same failure was not observed in later sampled runs.

Greedy decoding may have contributed, but this experiment does not prove that it caused every loop or that 0.7 is optimal for every model. Reproducibility requires recording the full sampling configuration and seed, not equating reproducibility with greedy decoding.

An Experimental Repetition Circuit Breaker

Sampling changes do not protect a shared inference server from every runaway request. I added an opt-in exact-repetition detector to the local llama.cpp package and submitted it as llama.cpp PR #26039. At the time of writing, the PR is open. In the proposed patch, the feature is disabled by default.

When enabled, the detector stores a bounded tail of generated token IDs and stops if an exact contiguous pattern repeats at the output tail. The default threshold values are:

min_pattern_size = 16
max_pattern_size = 1024
min_count        = 4

The local router enables it with:

[*]
repetition-detection = on

Unlike a repetition penalty, the detector does not alter logits or token probabilities. It changes only the stop decision. The server records:

stopped by repetition detection, n_decoded = ...

The OpenAI-compatible response uses finish_reason: "length"; native response metadata can retain stop_type: "repetition". A benchmark report that records only the OpenAI finish reason cannot distinguish this stop from the ordinary output limit, so the server log must be preserved and correlated with the case.

This detector is deliberately narrow. It does not detect semantic paraphrase loops, repair templates, or solve client cancellation. It can also stop legitimate output that intentionally repeats a sufficiently long exact token sequence. I have not yet measured its false-positive rate or decoding overhead across representative workloads. It should be treated as an experimental, observable circuit breaker rather than a general decoding fix.

Starting Protocol For The Final Aider Run

The final comparison started with the following Aider settings:

edit format:        diff
tries:              2
benchmark workers:  1
exercises:          all 225
request timeout:    1800 seconds
temperature:        0.7
max output tokens:  32768
Aider context metadata: 131072

The llama.cpp router started with:

[*]
parallel = 1
repetition-detection = on
cache-ram = 2048
ctx-size = 131072
n-predict = 32768
cache-type-k = q8_0
cache-type-v = q8_0
flash-attn = on

The package was llama.cpp-cuda b10092-1; the server reported version 10092, and the local repetition-detection package revision was commit 7e0c976.

benchmark workers=1 serializes Aider cases. parallel=1 independently limits llama.cpp to one active decoding slot. These values were chosen to avoid concurrent requests on this machine and make case-level timing easier to interpret. They are experimental controls, not requirements for running Aider.

Part 2 must still verify that the package, model artifacts, templates, and configuration remained unchanged until the run completed.

A Reproducible Result Needs More Than A Model Name

For each published result, I now preserve:

• model repository, exact GGUF filename, and quantization;
• llama.cpp package/revision and relevant build options;
• endpoint model ID, chat template, and template arguments;
• reasoning mode and final-answer extraction behavior;
• input/context and output limits;
• seed and complete sampling parameters;
• benchmark version, tasks, attempts, and edit format;
• benchmark-worker and server-slot concurrency;
• raw response, validator checks, execution result, latency, and token usage;
• timeout, output-limit, and server-side repetition events.

The last item matters because the same OpenAI-compatible finish reason can hide different server-side stop conditions.

Conclusions

The results support five limited conclusions:

1. Tool calling, strict serialization, debugging, and repository editing are distinct capabilities and should not be averaged into an unexplained score.
2. Reasoning models require an explicit output contract and enough budget to produce a final answer.
3. Exact-format checks should be reported separately from semantic and execution-backed checks.
4. Greedy decoding was present during the problematic Aider runs, but the retained evidence establishes correlation rather than universal causation.
5. Timeouts, output caps, and exact-repetition detection contain runaway output; they do not replace diagnosis of templates, parsers, sampling, or model behavior.

Part 2 will report the completed Aider Polyglot run with Pass@1, Pass@2, language-level results, elapsed time, malformed or rejected patches, test failures, timeouts, output-limit events, and server-side repetition stops. Only that protocol-fixed run will be used for coding-agent recommendations.