Working drawing 02 · Models← Benchmarks
165 models.
32 models tested with a written entry, why each was tried, and what came of it, and every other configuration the run log measured. The seats were settled on 20 September 2026: the models that won one come first; the rest are measured, not seated.
- Configurations
- 165
- With a write-up
- 32
- Scored runs
- 582
Measurements taken on an Apple M1 Max with 64 GB of unified memory.
Every number here comes from a locked dataset with a recorded grader. Where a measurement is not valid, it is marked rather than removed.
- JudgegenerateornithContradiction 94%. Confidence you can put a threshold on.Seated
- Taggemma4-12b93.8 and 97.9 on the tagger suites, best on the board.Seated
- Dispatchk2-3.7b100 of 100 tool calls. Refuses small talk by design.Seated
- Extractedge0 pool95% on field extraction from curated evidence.Seated
- Screengranite-guardian-4.1-8b100% on fabrication and misattribution screens.Seated
- Decidetemporalvon-1.0Which fact came last: 100% in 95 ms, on the CPU.Seated
- Decidealiaslaya-421mSame thing, different name: 100 of 100 in 42 ms, on the CPU.Seated
- Protectgliner + presidioEach catches what the other misses. Egress fails closed.Seated
- Embedbge-m3Native 8K window, dense and sparse.Seated
- Rerankbge-reranker-v2-m3A 13-point margin on the failure that broke recall in August.Seated
- Seephi4-visionRead a real invoice end to end. Swapped in on demand.Benched
The fleet's verb registry as settled on 20 September 2026 (OKF fleet-verbs), plus the seats recorded on the model cards in the benchmark export of the same date. Seats without a link are not language models and are measured outside this run log.
Tested as a cheap tagger after the 80B proved expensive for a task that runs constantly.
- Granite Guardian 4.1 8BSeated · Screen
8B · dense · Q4_K_M GGUF, and MLX mixed_3_6 at 3.8GB · 5.1 GB · resident
llama.cppmlxomlx
A purpose-built safety and screening classifier, tested for the judge seat where a general model was overkill.
- K2-Horizon-3.7B (IFM)Seated · Dispatch
3.7B · dense · Q4_K_M · 2.9 GB · resident
llama.cppllama.cpp (k2horizon fork)
The smallest model measured that perceives tool need. Held at three precisions (Q4/Q8/BF16) which made it the subject of the precision ladder.
- K2-Horizon-36B-A4B (IFM)Seated · agent
36B / 4B active · MoE · Q4_K_M · 20.8 GB · resident
llama.cpp (k2horizon fork)
MoVA (Mixture-of-Value Attention) extends sparsity into the value component of attention. 4B active gives it 25 tok/s at 5k where a dense 27B manages 8.6. Ties the top of the board on tool invocation and holds a 200-step chain without deviating.
- Qwen3.8 27BSeated · agentic
27B · dense · MLX 4-bit (MTP build, template patched to reasoning_effort=medium) · 15.0 GB · resident
llama.cppmlxomlx
The dense counterweight to the MoE fleet, and the model where MLX first beat llama.cpp. Promoted to the agent seat on 2026-09-09 after toolcall/1.
An agent-tuned MoE, tested on the tool suites where its specialism should show.
- BTL-4 (Q4_K_M)Measured, not seated
35.1B / 2.1B active · MoE · Q4_K_M (community conversion) · 20.5 GB · resident
llama.cpp
Fine-tuned from Ornith-1.0-35B on an execution-gated corpus: trajectories kept only where the code ran and passed. 256 experts, 8 routed, 262k native context, ~20 KB/token KV.
A coding model, tested to see whether coding tuning transfers to structured extraction.
The base model for the tagger fine-tune, and the smallest thing worth measuring.
Tested as a fast MoE with a 202k context window, and as the first model we quantised ourselves for MLX.
Tested because IBM builds explicitly for structured JSON output, and nullschema is the weakest axis across the whole fleet.
Tested as an extremely sparse small MoE, 1B active, to find the floor of useful.
Previously dismissed at 10.4 tok/s without ever being scored. Re-examined because it is multimodal, and scanned invoices are the part of the accountancy job nothing else can read.
A second non-Qwen coding candidate, tested alongside Devstral.
The first coding model measured against a real coding contract rather than an extraction suite.
The small reference and runtime test subject - the only model held in GGUF, MLX-native and BF16 simultaneously, which carried the whole runtime comparison. HumanEval 86.8% strict, beating qwen3.6-35b-a3b by being less verbose.
- Qwen3-Next-80B-A3B Instruct, REAP prunedMeasured, not seated
60B / ~3B active · MoE · IQ4_XS · 30.4 GB · resident
llama.cpp
Tested to find out whether expert pruning costs the extraction seat its abstention. The full model holds that seat at 40 GB, which is most of the memory budget on a 64 GB machine.
- Qwen3.6 35B-A3BMeasured, not seated
35B / 3B active · MoE · MLX 4-bit (also held as UD-Q5_K_M GGUF) · 19.0 GB · resident
llama.cppmlxomlx
Tested as the fast general MoE, and used as the daily coding driver in DSH. Moved to an MLX build on 2026-09-09.
- Qwen3.8-27B GSQ-RCO IQ3_S (ISTA-DASLab)Measured, not seated
27B · dense · IQ3_S (per-tensor gradient search) · 11.3 GB · resident
llama.cpp
Same weights as qwen3.8-27b at ~3 bits instead of ~4.5, with the quantisation type chosen per tensor by gradient search under a size budget. Retains the MTP head. 26% fewer bytes per token and 8% faster.
Full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-9B architecture, ~70k curated teacher traces. Tested as a cheap utility seat.
- Spark-X2.5-4B (XHToken)Measured, not seated
4B · dense · Q4_K_M · 2.4 GB · resident
llama.cpp (build 10893+)
Hybrid attention - one full-attention layer per three sliding-window layers - giving a native 1M context. Apache 2.0, 20T training tokens, marketed for agentic workflows with claimed native compatibility for Codex, Claude Code and Hermes.
- Ternary-Bonsai-27B (PQ2_0)Measured, not seated
26.9B · dense · PQ2_0 (2.13 bpw, group-128) · 7.2 GB · resident
llama.cpp (PrismML fork)
prism-ml's group-128 ternary pack, built for their own llama.cpp fork's hybrid-attention kernels. The upstream Q2_g64 variant lacks them.
- Ternary-Bonsai-27B (prism-ml)Measured, not seated
27B · dense · Q2_g64 (1.71 bits/weight) · 7.1 GB · resident
llama.cpp
End-to-end ternary weights on a Qwen3.6-27B backbone with ~75% linear attention, claiming 262K context and 94.6% of FP16 intelligence retained. 11.7 GB resident at a 128k window - a third of K2-36B for the same context.
- Bonsai-27B (Q1_0)Rejected
26.9B · dense · Q1_0 (1.13 bpw) · 3.5 GB · resident
llama.cppllama.cpp (PrismML fork)
Same base as the ternary build at a more aggressive operating point. Tested to find which capabilities survive 1-bit.
- Bonsai-8B (Q1_0)Rejected
8.0B · dense · Q1_0 (1.13 bpw) · 1.1 GB · resident
llama.cppllama.cpp (PrismML fork)
Derived from Qwen3-8B, not the Qwen3.6-27B backbone the larger Bonsai models use. Tested as the small corner of a size-by-bit-width grid.
- BTL-4 Compact (IQ2_XXS)Rejected
35.1B / 2.1B active · MoE · IQ2_XXS (2.30 bpw) · 10.0 GB · resident
llama.cpp
The only build badtheorylabs publish. Tested to see what an aggressive IQ codebook costs on a sparse MoE.
Tested to see whether reasoning improves extraction, on the assumption that thinking helps.
A custom tagger, fine-tuned with MLX LoRA on 270 turns, intended to replace the general model in the tagging seat.
Tested as a fast MoE, 45.9 tok/s, and briefly the best raw scorer on some suites.
The incumbent. Chosen because 3B active parameters give MoE speed at 80B quality, and it fits at 40GB with room for a long context.
Tested as the largest MoE that would fit, on the assumption that more parameters would win.
Tested because the M64 shard layout claims to run a 180B model on 64GB by keeping a 51B n-gram embedding table on SSD.
- edge0-35bSeated · Extract
llama.cpp
- k2-3.7bSeated · Dispatch
llama.cpp
- Measured only130 more configurations the run log measured, without a write-up. Each has a page with its runs.
- agents-a1-35b-b10618no write-up
llama.cpp
- bonsai2-27b-pq2no write-up
llama.cpp
- bonsai2-27b-ptq1no write-up
llama.cpp
- btl-4-q4km-recheckno write-up
llama.cpp
- edge0-35b-recheckno write-up
llama.cpp
- flashnext-4.27bpw-m64no write-up
llama.cpp
- gemma4-12b-abno write-up
llama.cpp
- gemma4-12b-b10618no write-up
llama.cpp
- gemma4-12b-b10893no write-up
llama.cpp
- gemma4-12b-midrangeno write-up
llama.cpp
- gemma4-12b-r1no write-up
llama.cpp
- gemma4-12b-r2no write-up
llama.cpp
- gemma4-12b-r3no write-up
llama.cpp
- gemma4-12b-recheckno write-up
llama.cpp
- gemma4-e2b-abno write-up
llama.cpp
- gemma4-e2b-b10618no write-up
llama.cpp
- gemma4-e2b-r1no write-up
llama.cpp
- gemma4-e2b-r2no write-up
llama.cpp
- gemma4-e2b-r3no write-up
llama.cpp
- gemma4-e4bno write-up
llama.cpp
- gemma4-e4b-b10618no write-up
llama.cpp
- gemma4-e4b-r1no write-up
llama.cpp
- gemma4-e4b-r2no write-up
llama.cpp
- gemma4-e4b-r3no write-up
llama.cpp
- glm-4.7-flash-q40no write-up
llama.cpp
- gpt-oss-20bno write-up
llama.cpp
- gpt-oss-20b-study-defaultno write-up
llama.cpp
- granite-4.2-30bno write-up
llama.cpp
- granite-guardianno write-up
llama.cpp
- granite-guardian-3.1-2bno write-up
llama.cpp
- k2-3.7b-q4-precisionno write-up
llama.cpp
- k2-3.7b-q8-precisionno write-up
llama.cpp
- k2-36b-a4b-q4no write-up
llama.cpp
- k2-36b-ctx128k-q8no write-up
llama.cpp
- k2-36b-ctx32kno write-up
llama.cpp
- k2-36b-extractionno write-up
llama.cpp
- k2-36b-np1no write-up
llama.cpp
- k2-36b-q4_0no write-up
llama.cpp
- k2-36b-Q4_1no write-up
llama.cpp
- k2-36b-Q5_0no write-up
llama.cpp
- k2-36b-Q5_1no write-up
llama.cpp
- k2-36b-q8-precisionno write-up
llama.cpp
- k2-36b-vendor-k2no write-up
llama.cpp
- k2-36b-vendor-k2bno write-up
llama.cpp
- k2-4b-bf16-precisionno write-up
llama.cpp
- k2-7bno write-up
llama.cpp
- k2-7b-bf16-precisionno write-up
llama.cpp
- k2-7b-q4-precisionno write-up
llama.cpp
- k2-7b-q8-precisionno write-up
llama.cpp
- k2-horizon-7bno write-up
llama.cpp
- lfm25-8b-a1b-midrangeno write-up
llama.cpp
- minicpm5-2bno write-up
llama.cpp
- minicpm5-2b-exl3no write-up
llama.cpp
- minicpm5-2b-q8-nojinjano write-up
llama.cpp
- minicpm5-2b-q8-np1no write-up
llama.cpp
- minicpm5-2b-q8-np4-todayno write-up
llama.cpp
- neohorse-1-4bno write-up
llama.cpp
- ornith-1.5-9bno write-up
llama.cpp
- ornith-35b-taggerno write-up
llama.cpp
- ornith-9b-kv-f16no write-up
llama.cpp
- ornith-9b-kv-q40no write-up
llama.cpp
- ornith-9b-kvq4no write-up
llama.cpp
- q36-toolparserno write-up
llama.cpp
- qwen-agentworld-35bno write-up
llama.cpp
- qwen3-8b-midrangeno write-up
llama.cpp
- qwen3-coder-30b-b10618no write-up
llama.cpp
- qwen3-next-80b-a3b-iq4xsno write-up
llama.cpp
- qwen3-next-80b-agno write-up
llama.cpp
- qwen3-next-80b-coderno write-up
llama.cpp
- qwen3-next-80b-reap-agno write-up
llama.cpp
- qwen3-next-80b-tagno write-up
llama.cpp
- qwen3.5-4b-baseno write-up
llama.cpp
- qwen3.5-9b-q8no write-up
llama.cpp
- qwen3.5-9b-q8-b10618no write-up
llama.cpp
- qwen3.6-35b-agno write-up
llama.cpp
- qwen3.6-35b-swiftlmno write-up
llama.cpp
- qwen3.6-35b-tagno write-up
llama.cpp
- qwen3.8-27b-noreasonno write-up
llama.cpp
- qwen3.8-27b-q4_0no write-up
llama.cpp
- qwen3.8-iq1sno write-up
llama.cpp
- qwen35-9b-gguf-q40no write-up
llama.cpp
- qwen35-9b-gguf-q4kmno write-up
llama.cpp
- qwen38-27b-np1no write-up
llama.cpp
- qwen38-27b-q3kxl-chadno write-up
llama.cpp
- qwen38-gsqrco-iq3sno write-up
llama.cpp
- qwen38-rebase-b10893no write-up
llama.cpp
- qwen38-vendor-qwen-thinkno write-up
llama.cpp
- reap-60bno write-up
llama.cpp
- reap-60b-20260910no write-up
llama.cpp
- reap-60b-4bit-reprono write-up
llama.cpp
- reap-60b-gguf-q4kmno write-up
llama.cpp
- spark-x25-4bno write-up
llama.cpp
- tagger-e2b-v1-dl-tagger-e2b-v1no write-up
llama.cppmlx
- ternary-bonsai-8bno write-up
llama.cpp
By outcome
What happened after testing, as of 20 September 2026. Rejected and removed entries are kept because the reason is the useful part.
Measured, not seated 18
Agents-A1 35B, BTL-4 (Q4_K_M), Devstral Small 2 24B, Gemma 4 E2B, GLM-4.7-Flash, Granite 4.2 30B, LFM 2.5 8B-A1B, Muse-Glimmer 30B, Ornith 1.5 35B, Qwen3 Coder 30B, Qwen3-8B, Qwen3-Next-80B-A3B Instruct, REAP pruned, Qwen3.6 35B-A3B, Qwen3.8-27B GSQ-RCO IQ3_S (ISTA-DASLab), Qwen3.8-9B-Distill (Q8_0), Spark-X2.5-4B (XHToken), Ternary-Bonsai-27B (PQ2_0), Ternary-Bonsai-27B (prism-ml)
Rejected 5
Bonsai-27B (Q1_0), Bonsai-8B (Q1_0), BTL-4 Compact (IQ2_XXS), Qwen3-Next-80B-A3B Thinking, tagger-e2b-v1 (LoRA fine-tune)
By architecture
Dense models stream every weight per token and are bandwidth bound. Mixture of experts models read only their active experts, so a 35B MoE can run five times faster than a 30B dense model.
dense 17
Bonsai-27B (Q1_0), Bonsai-8B (Q1_0), Devstral Small 2 24B, Gemma 4 12B, Gemma 4 E2B, Granite 4.2 30B, Granite Guardian 4.1 8B, K2-Horizon-3.7B (IFM), Muse-Glimmer 30B, Qwen3-8B, Qwen3.8 27B, Qwen3.8-27B GSQ-RCO IQ3_S (ISTA-DASLab), Qwen3.8-9B-Distill (Q8_0), Spark-X2.5-4B (XHToken), tagger-e2b-v1 (LoRA fine-tune), Ternary-Bonsai-27B (PQ2_0), Ternary-Bonsai-27B (prism-ml)
MoE 15
Agents-A1 35B, BTL-4 (Q4_K_M), BTL-4 Compact (IQ2_XXS), GLM-4.7-Flash, K2-Horizon-36B-A4B (IFM), LFM 2.5 8B-A1B, Nemotron 3.5 Lightning 30B, Ornith 1.5 35B, Qwen3 Coder 30B, Qwen3-Next-80B-A3B Instruct, Qwen3-Next-80B-A3B Instruct, REAP pruned, Qwen3-Next-80B-A3B Thinking, Qwen3.5 122B-A10B, Qwen3.6 35B-A3B, Qwen3.8-Flash-Next
By residency
Whether the weights must sit in unified memory. One model here runs only because it does not: the M64 layout keeps a 51B embedding table on SSD.
resident 28
Agents-A1 35B, Bonsai-27B (Q1_0), Bonsai-8B (Q1_0), BTL-4 (Q4_K_M), BTL-4 Compact (IQ2_XXS), Devstral Small 2 24B, Gemma 4 12B, Gemma 4 E2B, GLM-4.7-Flash, Granite 4.2 30B, Granite Guardian 4.1 8B, K2-Horizon-3.7B (IFM), K2-Horizon-36B-A4B (IFM), LFM 2.5 8B-A1B, Muse-Glimmer 30B, Ornith 1.5 35B, Qwen3 Coder 30B, Qwen3-8B, Qwen3-Next-80B-A3B Instruct, REAP pruned, Qwen3-Next-80B-A3B Thinking, Qwen3.6 35B-A3B, Qwen3.8 27B, Qwen3.8-27B GSQ-RCO IQ3_S (ISTA-DASLab), Qwen3.8-9B-Distill (Q8_0), Spark-X2.5-4B (XHToken), tagger-e2b-v1 (LoRA fine-tune), Ternary-Bonsai-27B (PQ2_0), Ternary-Bonsai-27B (prism-ml)
By runtime
Which engines the model was measured under. Two suites cannot run under MLX at all, because its server ignores schema constraints.
llama.cpp 25
Agents-A1 35B, BTL-4 (Q4_K_M), BTL-4 Compact (IQ2_XXS), Devstral Small 2 24B, Gemma 4 12B, Gemma 4 E2B, GLM-4.7-Flash, Granite 4.2 30B, Granite Guardian 4.1 8B, LFM 2.5 8B-A1B, Muse-Glimmer 30B, Nemotron 3.5 Lightning 30B, Ornith 1.5 35B, Qwen3 Coder 30B, Qwen3-8B, Qwen3-Next-80B-A3B Instruct, Qwen3-Next-80B-A3B Instruct, REAP pruned, Qwen3-Next-80B-A3B Thinking, Qwen3.5 122B-A10B, Qwen3.6 35B-A3B, Qwen3.8 27B, Qwen3.8-27B GSQ-RCO IQ3_S (ISTA-DASLab), Qwen3.8-9B-Distill (Q8_0), Qwen3.8-Flash-Next, Ternary-Bonsai-27B (prism-ml)
mlx 11
Gemma 4 E2B, GLM-4.7-Flash, Granite 4.2 30B, Granite Guardian 4.1 8B, Nemotron 3.5 Lightning 30B, Ornith 1.5 35B, Qwen3-8B, Qwen3-Next-80B-A3B Instruct, Qwen3.6 35B-A3B, Qwen3.8 27B, tagger-e2b-v1 (LoRA fine-tune)
llama.cpp (PrismML fork) 3
Bonsai-27B (Q1_0), Bonsai-8B (Q1_0), Ternary-Bonsai-27B (PQ2_0)
llama.cpp (k2horizon fork) 2
llama.cpp (build 10893+) 1
By family
Related models tested from the same lineage. Comparing within a family isolates the change.
bonsai 4
Bonsai-27B (Q1_0), Bonsai-8B (Q1_0), Ternary-Bonsai-27B (PQ2_0), Ternary-Bonsai-27B (prism-ml)
qwen3-next 3
Qwen3-Next-80B-A3B Instruct, Qwen3-Next-80B-A3B Instruct, REAP pruned, Qwen3-Next-80B-A3B Thinking