FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Model← Models

Qwen3 Coder 30B

30B mixture of experts, 3B active, Q4_K_M, 18.0 GB.

Parameters
30B / 3B active
Quant
Q4_K_M
Size
18.0 GB
Family
qwen3.x
RecordStatus · runtime · seat

qwen3-coder-30b

Measured, not seatedMoEresidentllama.cpp

Model card ↗

Outcome as of 20 September 2026, from the seat registry. The write-up below was made when the model was tested.

Why it was tested

The first coding model measured against a real coding contract rather than an extraction suite.

What came of it

0/12 strict on shapegen but a mean of 0.84, blocked by one rule: it never emits @scope on any case. On sitebuild it wrote 13 files while running the build twice, then exhausted its budget with a broken tree. A workflow failure, not a knowledge gap.

Recorded runs3 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
grounding77.6%69.5–84.0Passed 97/125llama.cppextraction2026-08-24
shapegen0.0%0.0–24.3Passed 0/12llama.cppextraction2026-09-07
sitebuild0.0%0.0–79.3Passed 0/1llama.cppagentic-fast2026-09-07

Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Every run log entry for this model (3)
FinishedSuiteRuntime · profileScoreCoverageGrader · dataset · config
shapegenshapegen/2llama.cppextraction0.0%Coverage 12/12g f23222ad8df22fd8d fc27d4e1a999c675c cbf6852b8edc0984
sitebuildsitebuild/1llama.cppagentic-fast0.0%Coverage 1/1g b2c603d6c0d6afb3d 8b54725d2778dc60c 0cac83c8fbc03eec
groundinggrounding/1llama.cppextraction77.6%Coverage 125/125g 30b933dcc6a7e2d4d dcb396faf8b83b0ac ade06bde17b71832
FamilyOthers tested from the qwen3.x family
  • Qwen3.5 122B-A10BRemoved

    122B · MoE · IQ3 · 45.0 GB

    Tested as the largest MoE that would fit, on the assumption that more parameters would win.

  • Qwen3.6 35B-A3BMeasured, not seated

    35B · MoE · MLX 4-bit (also held as UD-Q5_K_M GGUF) · 19.0 GB

    Tested as the fast general MoE, and used as the daily coding driver in DSH. Moved to an MLX build on 2026-09-09.

  • Qwen3.8 27BSeated · agentic

    27B · dense · MLX 4-bit (MTP build, template patched to reasoning_effort=medium) · 15.0 GB

    The dense counterweight to the MoE fleet, and the model where MLX first beat llama.cpp. Promoted to the agent seat on 2026-09-09 after toolcall/1.

  • Qwen3.8-Flash-NextRemoved

    180B · MoE · AD-3.84bpw M64 · 79.1 GB

    Tested because the M64 shard layout claims to run a 180B model on 64GB by keeping a 51B n-gram embedding table on SSD.