Eight Quantized Local Models on Aider Polyglot: An Operational Snapshot

Part 1 describes the benchmark design and the runtime failures that had to be resolved before collecting these results: Debugging Local LLM Benchmarks.

Snapshot status, July 27, 2026: 8 models have complete 225-exercise runs and are included in the aggregate tables. All scheduled full-batch runs are complete.

This report summarizes eight quantized GGUF artifacts in one local Aider setup. Each artifact was evaluated on all 225 Aider Polyglot exercises with the same diff edit format and a maximum of two attempts per exercise. The primary outcome is the number of exercises whose tests passed by the end of the second attempt.

This is not a controlled model leaderboard. The runner controls were fixed, but the long-running batches were resumed over several days while the local serving stack was being maintained. In particular, the active AgentWorld chat template changed after 10 of its 225 result files had been produced. Some later results also span llama.cpp package updates. The aggregate rows remain useful as an operational record of this system, but they cannot isolate model quality from serving configuration.

The highest aggregate Pass@2 values were 70.2% for Qwen AgentWorld 35B A3B and 69.3% for Ornith 1.0 35B. That two-exercise difference does not establish a stable ordering: each artifact was run once, generation was stochastic, and no repeated-seed experiment was performed. AgentWorld’s mixed-template run adds a separate confound. The defensible conclusion is limited to the recorded rows: the two runs produced similar aggregate solve counts but different resource and failure profiles.

Protocol and measurement

The benchmark used the Aider Polyglot dataset at commit 7e0611e: 225 Exercism-derived tasks across C++, Go, Java, JavaScript, Python, and Rust. For each task, Aider supplied repository context, requested an edit, applied any parseable patch, and ran the task’s tests.

The local patch adds a --timeout option and forwards it to LiteLLM. It changes request containment, not the benchmark’s scoring logic. The Aider checkout is therefore recorded as 5dc9490-dirty. The patch is not publicly archived, so the upstream commit alone is insufficient to reproduce the runner.

All local runs used one OpenAI-compatible llama.cpp endpoint and executed serially. The endpoint URL and runner parameters were stable; the underlying server package and some model-template configuration were not frozen for the entire collection. Wall time includes model generation, Aider processing, patch application, and tests. It is not a pure inference-throughput metric.

The metrics used below are:

• Pass@1: tests passed after the first attempt.
• Pass@2: tests passed after either the first or second attempt.
• Malformed case: at least one Aider num_malformed_responses event was recorded for the exercise.
• No malformed event: the complement of malformed cases. This does not imply that the patch applied or passed tests.
• Completion tokens / solved: total recorded completion tokens for the entire run divided by its Pass@2 solve count.
• Seconds / solved: total recorded wall time for the entire run divided by its Pass@2 solve count.

Pass@2 is an agent-loop metric, not the statistical pass@k estimator used for independent code samples. Because there is one run per artifact, the tables report observed outcomes only; they do not estimate run-to-run variance.

The exporter reads each native .aider.results.json and retains one row per completed exercise. A watchdog timeout sidecar contributes to handled coverage but not to the Pass@1 or Pass@2 denominator unless a result file exists. The export contains 225 result files for every included artifact and no exporter warnings. Completeness here means complete task coverage, not a claim that the serving environment was immutable.

Evaluated artifacts

The measured unit is the complete stack: checkpoint, quantization, chat template, llama.cpp serving configuration, Aider, and edit protocol. Because parts of that stack changed during collection, the table is descriptive rather than a controlled comparison. The results must not be attributed to the unquantized model families alone.

Qwen AgentWorld needs a role qualification. Its model card and technical report describe a language world model that predicts environment state from interaction history and agent actions. Here it was used directly as Aider’s coding model. Its aggregate row also combines 10 cases produced before an active-template change with 215 produced afterward. The result is therefore exploratory evidence of transfer to the coding-actor role, not a clean estimate for one AgentWorld serving configuration.

Observed aggregate results

Only complete 225-exercise runs appear in this table.

AgentWorld and Ornith differed by two Pass@2 solves. Their task-level outcomes were not interchangeable: both solved 126 exercises, AgentWorld alone solved 32, Ornith alone solved 30, and neither solved 37. This paired breakdown is more informative than the 0.9-point aggregate gap. It still does not support a model ordering because the runs were neither repeated nor configuration-matched.

The recorded resource difference is larger. Relative to Ornith, AgentWorld consumed 58% more completion tokens and 55% more total wall time while solving two additional exercises. Qwen3 Coder Next solved 145 exercises with substantially less recorded output and wall time than either top row. These are properties of the recorded runs, not normalized model-efficiency estimates.

All scheduled full-batch models have complete 225-exercise result sets.

The chart visualizes the same local Pass@1 and Pass@2 values as the canonical table above.

Cloud Reference: GPT-5.4 Mini Without Reasoning

One hosted run provides a reference point, not a ninth row in the local comparison. GPT-5.4 mini used the OpenAI API, reasoning_effort=none, the same diff format, and at most two attempts. Aider recorded zero thinking tokens and approximately $3.53 in API cost.

Its latency is not comparable with local serial inference, and its no-reasoning configuration is not a matched counterpart to the local serving profiles. The run establishes only the result of this budget-constrained API configuration. Any claim about the effect of reasoning would require a paired run with reasoning enabled.

Contribution of the second attempt

Every artifact gained at least 21.8 percentage points from the second attempt. The gain belongs to the combined model-Aider loop: the second request includes feedback from the first patch and its tests. It does not measure two independent samples from the original prompt.

For deployment, both columns matter. Pass@1 approximates how often one model round is sufficient; Pass@2 measures the tested repair loop. Reporting only Pass@2 hides the additional requests, tokens, and time required to reach it.

Recorded solve rate, wall time, and output volume

No artifact dominates all three dimensions.

• AgentWorld recorded 158 solves, 513.3 seconds per case, and 4.20 million completion tokens.
• Ornith recorded 156 solves, 331.2 seconds per case, and 2.66 million completion tokens.
• Qwen3 Coder Next recorded 145 solves, 256.7 seconds per case, and 1.09 million completion tokens.
• Qwen3.6 Unsloth and Aggressive were the fastest local runs at 88.1 and 82.2 seconds per case, with 117 and 112 solves respectively.
• Both Gemma4 artifacts combined lower solve counts with high malformed-case rates; their wall time was not offset by higher Pass@2.

These values are descriptive run totals. Since the serving stack was not frozen throughout collection, they should not be used as a controlled speed ranking or extrapolated to other hardware. GPT-5.4 mini is excluded because its hosted execution path is different.

This chart visualizes recorded Pass@2 and median wall time. The serving configuration was not frozen across all runs, so it is not a controlled speed ranking. Bubble size encodes completion tokens per solved exercise.

Results by programming language

The language groups contain different numbers of exercises, so counts and percentages are reported together.

The largest observed per-language value is Qwen3 Coder Next’s 28/34 on Python. AgentWorld and Ornith are more even across the six groups. These subgroup sizes are too small for broad claims about language competence, and the tasks are not matched for difficulty across languages.

Protocol failures

Malformed output is separate from test failure. A syntactically parseable edit can apply and still fail tests; a malformed response can prevent Aider from applying the intended edit at all.

In these artifacts, num_malformed_responses is Aider’s protocol-level counter. It includes failures such as SEARCH/REPLACE blocks that cannot be matched; it does not mean that the OpenAI-compatible HTTP response contained invalid JSON.

Ornith recorded no malformed-response events, although five exercises exhausted context. Qwen3.6 Unsloth had the lowest malformed-case rate after Ornith at 4.9%. The Gemma4 variants were incompatible with this fixed diff setup in 51.6% and 74.7% of cases respectively. That result is specific to the tested artifacts, templates, and edit format; it does not measure their performance with a model-specific format.

The aggregate export does not classify every patch rejection or valid-patch test failure. Such a taxonomy requires transcript-level analysis and is not inferred here.

The failure-rate chart summarizes protocol reliability; the adjacent table remains the exact accessible source.

Token and time efficiency

The following two ratio columns divide total run resources by the number of Pass@2 solves. They are run-level cost ratios, not conditional means over only the successful exercises.

Qwen3.6 Unsloth has the lowest completion-token ratio per solve; Qwen3.6 Aggressive has the lowest wall-time ratio. AgentWorld’s solve count is highest, but its completion-token ratio is 56% above Ornith’s and 254% above Qwen3 Coder Next’s. These ratios capture operational cost under the measured stack, not architectural efficiency or energy use.

Excluded qualification run

One candidate was stopped during qualification after an observed pass rate of approximately 0.1. It did not enter the full-run roster or aggregate table.

Full batch complete

All scheduled full-batch models have 225 completed exercise results.

Comparison with BenchLocal

BenchLocal and Aider Polyglot measure different system behavior. BenchLocal is a set of short, deterministic checks for structured output, instruction following, tool selection, extraction, and debugging. Aider Polyglot measures repository edits validated by tests through a particular text-edit protocol.

The observed rank changes are substantial. Gemma4 Unsloth placed second in BenchLocal but seventh here, where 51.6% of cases contained a malformed-response event. Qwen3 Coder Next placed sixth in BenchLocal and third here. Agents-A1 moved from eighth to fourth. These reversals show that short-form contract compliance is not a substitute for testing a repository-editing loop. They are not evidence for a precise cross-benchmark ranking, especially given the serving confound in the Aider collection.

They do not show which benchmark is more correct. Each result is conditional on its task distribution, parser, prompts, and scoring rules.

Candidates for a controlled follow-up

The snapshot is sufficient to choose candidates for a stricter rerun, not to select a production default:

• Ornith 1.0 35B should be retained because its run combined 156 solves with no malformed-response cases. Five cases exhausted context.
• Qwen3 Coder Next should be retained because its run reached 145/225 with lower output than the two highest-scoring rows. It recorded a 9.8% malformed-case rate.
• Qwen3.6 35B A3B Unsloth provides a lower-cost comparison point: 117/225 solves and 4.9% malformed cases in the recorded run.
• Agents-A1 APEX is relevant for testing the trade-off between its 130/225 solve count and its 17.8% malformed-case rate.
• Qwen3.6 Aggressive adds little evidence beyond the Unsloth artifact: it solved five fewer tasks and had more than twice the malformed-case rate.
• The tested Gemma4 artifacts should not be rerun unchanged under diff. A follow-up would need a justified protocol or template change declared in advance.
• AgentWorld requires a clean rerun from zero with one frozen template and server build before its aggregate score can be compared with the other rows.

A controlled follow-up should pin the llama.cpp build and configuration, checksum every template, start new run directories, record a seed where the backend supports it, and repeat each selected artifact across multiple seeds.

Limitations and data availability

The report does not support claims beyond the recorded runs:

1. The serving environment was not immutable. AgentWorld crossed an active chat-template change after 10 cases, and some resumed runs crossed llama.cpp package updates. This prevents a controlled model-to-model comparison.
2. Each artifact was run once at temperature 0.7. There are no repeated seeds, confidence intervals, or estimates of run-to-run variance.
3. The benchmark is a fixed set of 225 tasks, not a random sample of all software-engineering work.
4. The artifacts differ in checkpoint, quantization, template, architecture, and reasoning representation. The comparison does not isolate any one of those factors.
5. The fixed diff protocol measures both coding ability and compatibility with Aider’s parser. No per-model format tuning was performed.
6. Wall time includes the complete Aider exercise path and serial execution. It is not pure inference time.
7. Completion-token counts are those reported through Aider and LiteLLM. Server-side handling of hidden or tagged reasoning may differ between templates, so the counts are operational billing-style measurements rather than direct measures of internal computation.
8. The failure counters overlap at the case level and do not provide a complete causal taxonomy.
9. GPT-5.4 mini used a hosted API with reasoning disabled. Its latency, cost, and solve rate are not directly comparable to the local rows.

The raw CSV files, per-case results, runner patch, and native Aider artifacts are not currently public. The aggregate tables therefore cannot be independently audited from this article alone. Until a versioned archive is published, this report should be treated as a documented operational snapshot rather than a reproducible benchmark release.

The snapshot identifies three useful signals: second attempts changed every aggregate result materially; edit-protocol failure rates varied from 0% to 74.7%; and similar solve counts could require very different amounts of output and wall time. It does not establish a model leaderboard. The next experiment must freeze and publish the full serving configuration before using these rows to make a deployment decision.