Suite · nullschema← Suites
Schema shape and abstention
Is it the schema that causes fabrication, or is it forbidding absence?
- Cases
- 72
- Scored runs
- 79
- Models
- 79
- Top score
- 97.2%
SpecificationMethod · grading · caveat
- Method
- Three arms run over the same documents. In forbid, the field is required and non-nullable. In permit, the type union includes null. In optional, the key may be omitted. Only cases whose true answer is absent are used, because a case with a present answer cannot separate the arms.
- Grading
- A case passes when the model declines, either by returning null or by omitting the key. Returning any value counts as a failure and is further classified as misattribution or fabrication.
- Caveat
- The three arms must differ. If they come out identical, the server ignored the schema and the run is void.
- Version
nullschema/1- Cases
- 72
Recorded runs79 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite Qwen3-Next-80B-A3B Instruct 97.2% 95% interval 90.4–99.2
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| Qwen3-Next-80B-A3B Instructqwen3-next-80b | 97.2% | 90.4–99.2 | Passed 70/72 | llama.cppextraction | 2026-09-05 |
| qwen3-next-80b-a3b-iq4xs | 97.2% | 90.4–99.2overlaps the top interval | Passed 70/72 | llama.cppextraction | 2026-08-26 |
| Qwen3-Next-80B-A3B Instruct, REAP prunedqwen3-next-80b-reap | 97.2% | 90.4–99.2overlaps the top interval | Passed 70/72 | llama.cppextraction | 2026-09-08 |
| qwen3-next-80b-mlx-4bitunenforced | 95.8% | 88.5–98.6overlaps the top interval | Passed 69/72 | mlxextraction | 2026-09-07 |
| Granite 4.2 30Bgranite-4.2-30b-gguf | 90.3% | 81.3–95.2overlaps the top interval | Passed 65/72 | llama.cppextraction | 2026-09-08 |
| Gemma 4 12Bgemma4-12b | 88.9% | 79.6–94.3overlaps the top interval | Passed 64/72 | llama.cppextraction | 2026-08-24 |
| gemma4-12b-b10618 | 88.9% | 79.6–94.3overlaps the top interval | Passed 64/72 | llama.cppextraction | 2026-08-25 |
| gemma4-12b-midrange | 87.5% | 77.9–93.3overlaps the top interval | Passed 63/72 | llama.cppextraction | 2026-09-10 |
| reap-60b-mlx-4bit-uniformunenforced | 84.7% | 74.7–91.2overlaps the top interval | Passed 61/72 | mlxextraction | 2026-09-11 |
| qwen38-27b-np1 | 76.4% | 65.4–84.7 | Passed 55/72 | llama.cppextraction | 2026-09-10 |
| qwen38-rebase-b10893 | 76.4% | 65.4–84.7 | Passed 55/72 | llama.cppextraction | 2026-09-10 |
| gpt-oss-20b | 72.2% | 61.0–81.2 | Passed 52/72 | llama.cppextraction | 2026-09-12 |
| k2-36b-q8-precision | 70.8% | 59.5–80.1 | Passed 51/72 | llama.cppextraction | 2026-09-10 |
| qwen3.8-27b-q4_0 | 69.4% | 58.0–78.9 | Passed 50/72 | llama.cppextraction | 2026-09-11 |
| k2-36b-q4_0 | 65.3% | 53.8–75.2 | Passed 47/72 | llama.cppextraction | 2026-09-11 |
| Devstral Small 2 24Bdevstral-small-2-24b-b10618 | 62.5% | 51.0–72.8 | Passed 45/72 | llama.cppextraction | 2026-08-25 |
| ornith-1.5-35b-mlx-4bitunenforced | 62.5% | 51.0–72.8 | Passed 45/72 | mlxextraction | 2026-09-08 |
| qwen38-gsqrco-iq3s | 62.5% | 51.0–72.8 | Passed 45/72 | llama.cppextraction | 2026-09-10 |
| Qwen3.6 35B-A3Bqwen3.6-35b | 61.1% | 49.6–71.5 | Passed 44/72 | llama.cppextraction | 2026-08-27 |
| glm-4.7-flash-mlx-q8unenforced | 59.7% | 48.2–70.3 | Passed 43/72 | mlxextraction | 2026-09-08 |
| qwen35-9b-mlx-affine-g64unenforced | 59.7% | 48.2–70.3 | Passed 43/72 | mlxextraction | 2026-09-11 |
| flashnext-4.27bpw-m64 | 58.3% | 46.8–69.0 | Passed 42/72 | llama.cppextraction | 2026-09-08 |
| k2-36b-Q5_1 | 58.3% | 46.8–69.0 | Passed 42/72 | llama.cppextraction | 2026-09-11 |
| Qwen3-Next-80B-A3B Thinkingqwen3-next-80b-thinking | 58.3% | 46.8–69.0 | Passed 42/72 | llama.cppextraction | 2026-08-27 |
| k2-36b-Q5_0 | 56.9% | 45.4–67.7 | Passed 41/72 | llama.cppextraction | 2026-09-11 |
| k2-36b-vendor-k2b | 56.9% | 45.4–67.7 | Passed 41/72 | llama.cppvendor-k2b | — |
| ornith-9b-kvq4 | 56.9% | 45.4–67.7 | Passed 41/72 | llama.cppextraction | 2026-09-12 |
| qwen3-8b-mlx-8bitunenforced | 56.9% | 45.4–67.7 | Passed 41/72 | mlxextraction | 2026-09-08 |
| qwen3-next-80b-coder | 56.9% | 45.4–67.7 | Passed 41/72 | llama.cppextraction | 2026-08-27 |
| k2-36b-ctx32k | 54.2% | 42.7–65.2 | Passed 39/72 | llama.cppextraction | 2026-09-09 |
| k2-36b-extraction | 54.2% | 42.7–65.2 | Passed 39/72 | llama.cppextraction | 2026-09-09 |
| ornith-1.5-9b | 52.8% | 41.4–63.9 | Passed 38/72 | llama.cppextraction | 2026-09-12 |
| k2-36b-Q4_1 | 50.0% | 38.7–61.3 | Passed 36/72 | llama.cppextraction | 2026-09-11 |
| Qwen3.8-Flash-Nextqwen38-flash-next | 50.0% | 38.7–61.3 | Passed 36/72 | llama.cppextraction | 2026-09-05 |
| GLM-4.7-Flashglm-4.7-flash-q8 | 48.6% | 37.4–59.9 | Passed 35/72 | llama.cppextraction | 2026-08-26 |
| spark-x25-4b | 48.6% | 37.4–59.9 | Passed 35/72 | llama.cppextraction | 2026-09-10 |
| k2-36b-vendor-k2 | 47.2% | 36.1–58.6 | Passed 34/72 | llama.cppvendor-k2 | — |
| reap-60b-20260910 | 47.2% | 36.1–58.6 | Passed 34/72 | llama.cppextraction | 2026-09-10 |
| Qwen3.5 122B-A10Bqwen3.5-122b-a10b-iq3 | 45.8% | 34.8–57.3 | Passed 33/72 | llama.cppextraction | 2026-08-26 |
| qwen35-9b-mlx-affine-g32unenforced | 45.8% | 34.8–57.3 | Passed 33/72 | mlxextraction | 2026-09-11 |
| BTL-4 (Q4_K_M)btl-4-q4km | 44.4% | 33.5–55.9 | Passed 32/72 | llama.cppextraction | 2026-09-15 |
| glm-4.7-flash-q40 | 44.4% | 33.5–55.9 | Passed 32/72 | llama.cppextraction | 2026-09-12 |
| k2-3.7b-q8-precision | 43.1% | 32.3–54.6 | Passed 31/72 | llama.cppextraction | 2026-09-10 |
| k2-4b-bf16-precision | 43.1% | 32.3–54.6 | Passed 31/72 | llama.cppextraction | 2026-09-10 |
| K2-Horizon-3.7B (IFM)k2-horizon-3.7b | 43.1% | 32.3–54.6 | Passed 31/72 | llama.cppextraction | 2026-09-09 |
| glm-4.7-flash-mlx-q4unenforced | 40.3% | 29.7–51.8 | Passed 29/72 | mlxextraction | 2026-09-08 |
| agents-a1-35b-b10618 | 38.9% | 28.5–50.4 | Passed 28/72 | llama.cppextraction | 2026-08-26 |
| k2-3.7b-q4-precision | 38.9% | 28.5–50.4 | Passed 28/72 | llama.cppextraction | 2026-09-10 |
| k2-7b-bf16-precision | 37.5% | 27.2–49.0 | Passed 27/72 | llama.cppextraction | 2026-09-10 |
| k2-7b-q4-precision | 37.5% | 27.2–49.0 | Passed 27/72 | llama.cppextraction | 2026-09-10 |
| qwen3-coder-30b-b10618 | 37.5% | 27.2–49.0 | Passed 27/72 | llama.cppextraction | 2026-08-26 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 36.1% | 26.0–47.6 | Passed 26/72 | llama.cppextraction | 2026-09-15 |
| k2-7b-q8-precision | 36.1% | 26.0–47.6 | Passed 26/72 | llama.cppextraction | 2026-09-10 |
| ornith-35b-a3b-q40 | 36.1% | 26.0–47.6 | Passed 26/72 | llama.cppextraction | 2026-09-12 |
| k2-horizon-7b | 34.7% | 24.8–46.2 | Passed 25/72 | llama.cppextraction | 2026-09-09 |
| Ornith 1.5 35Bornith-1.5-35b-q6-b10618 | 34.7% | 24.8–46.2 | Passed 25/72 | llama.cppextraction | 2026-08-25 |
| qwen3-8b-midrange | 34.7% | 24.8–46.2 | Passed 25/72 | llama.cppextraction | 2026-09-10 |
| qwen35-9b-mlx-mxfp4unenforced | 34.7% | 24.8–46.2 | Passed 25/72 | mlxextraction | 2026-09-11 |
| qwen3.5-4b-base | 33.3% | 23.5–44.8 | Passed 24/72 | llama.cppextraction | 2026-09-13 |
| bonsai2-27b-pq2 | 31.9% | 22.3–43.4 | Passed 23/72 | llama.cppextraction | 2026-09-18 |
| bonsai2-27b-ptq1 | 31.9% | 22.3–43.4 | Passed 23/72 | llama.cppextraction | 2026-09-18 |
| minicpm5-2b | 31.9% | 22.3–43.4 | Passed 23/72 | llama.cppextraction | 2026-09-15 |
| qwen3.5-9b-q8-b10618 | 30.6% | 21.1–42.0 | Passed 22/72 | llama.cppextraction | 2026-08-25 |
| LFM 2.5 8B-A1Blfm2.5-8b-a1b | 27.8% | 18.8–39.0 | Passed 20/72 | llama.cppextraction | 2026-08-31 |
| lfm25-8b-a1b-midrange | 27.8% | 18.8–39.0 | Passed 20/72 | llama.cppextraction | 2026-09-10 |
| neohorse-1-4b | 27.8% | 18.8–39.0 | Passed 20/72 | llama.cppextraction | 2026-09-13 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 27.8% | 18.8–39.0 | Passed 20/72 | llama.cppextraction | 2026-09-14 |
| qwen35-9b-gguf-q4km | 23.6% | 15.3–34.6 | Passed 17/72 | llama.cppextraction | 2026-09-11 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 19.4% | 12.0–30.0 | Passed 14/72 | llama.cppextraction | 2026-09-14 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 18.1% | 10.9–28.5 | Passed 13/72 | llama.cppextraction | 2026-09-14 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 18.1% | 10.9–28.5 | Passed 13/72 | llama.cppextraction | 2026-09-14 |
| Nemotron 3.5 Lightning 30Bnemotron-lightning-30b | 15.3% | 8.8–25.3 | Passed 11/72 | llama.cppextraction | 2026-09-05 |
| gemma4-e4b | 13.9% | 7.7–23.7 | Passed 10/72 | llama.cppextraction | 2026-08-24 |
| gemma4-e4b-b10618 | 13.9% | 7.7–23.7 | Passed 10/72 | llama.cppextraction | 2026-08-25 |
| Gemma 4 E2Bgemma4-e2b | 12.5% | 6.7–22.1 | Passed 9/72 | llama.cppextraction | 2026-08-24 |
| gemma4-e2b-b10618 | 12.5% | 6.7–22.1 | Passed 9/72 | llama.cppextraction | 2026-08-25 |
| ternary-bonsai-8b | 11.1% | 5.7–20.4 | Passed 8/72 | llama.cppextraction | 2026-09-17 |
| nemotron-lightning-30b-mlx-4bitunenforced | 9.7% | 4.8–18.7 | Passed 7/72 | mlxextraction | 2026-09-08 |
| qwen35-9b-gguf-q40 | 6.9% | 3.0–15.2 | Passed 5/72 | llama.cppextraction | 2026-09-11 |
Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.
Flagged3 runs recorded under a flag
Runs recorded under a flagged label (void or killed part-way). They stay on the record and are not ranked.
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| granite-4.2-30b-mlx-q4flagged | 100.0% | 94.9–100.0 | Passed 72/72 | mlxextraction | 2026-09-08 |
| reap-60b-gguf-q4kmflagged | 100.0% | 94.9–100.0 | Passed 72/72 | llama.cppextraction | 2026-09-10 |
| qwen38-vendor-qwen-thinkflagged | 56.2% | 33.2–76.9 | Passed 9/16 | llama.cppvendor-qwen-think | 2026-09-10 |
Run log83 logged runs of this suite
Every logged run, newest first (83)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| bonsai2-27b-ptq1 | not recordedextraction | 31.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 2641ce03c27a545e | |
| bonsai2-27b-pq2 | not recordedextraction | 31.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 537688bcd022a4e2 | |
| ternary-bonsai-8b | not recordedextraction | 11.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 409ab3394165e872 | |
| minicpm5-2b | not recordedextraction | 31.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc a792a14d7d990d1b | |
| btl-4-q4km | not recordedextraction | 44.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 43c9232ce793d709 | |
| btl-4 | not recordedextraction | 36.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 9b2dcb073c569fce | |
| bonsai-27b-1bit | not recordedextraction | 18.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc b6bf04bcfeb45185 | |
| bonsai-8b-1bit | not recordedextraction | 19.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc de1a0f97f9063521 | |
| ternary-bonsai-27b | not recordedextraction | 27.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 004a863515b30c86 | |
| qwen3.8-9b-distill | not recordedextraction | 18.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 56c7dc8f982d59d3 | |
| qwen3.5-4b-base | not recordedextraction | 33.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 8ed6c2d36dcf6e58 | |
| neohorse-1-4b | not recordedextraction | 27.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc e55e57d062c38080 | |
| glm-4.7-flash-q40 | not recordedextraction | 44.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc a206ab6679bc0fed | |
| ornith-9b-kvq4 | not recordedextraction | 56.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 32a57cca965b6777 | |
| ornith-35b-a3b-q40 | not recordedextraction | 36.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 4324ac01a6993c44 | |
| ornith-1.5-9b | not recordedextraction | 52.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 0dd3a4ce0c1655b8 | |
| gpt-oss-20b | not recordedextraction | 72.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 262a6f5122bb9bbd | |
| qwen35-9b-mlx-mxfp4 | mlxextraction | 34.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc da30e30743dadebe | |
| qwen35-9b-mlx-affine-g32 | mlxextraction | 45.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 4cb05505e1d4b652 | |
| qwen35-9b-mlx-affine-g64 | mlxextraction | 59.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 987f7b78b76a6a60 | |
| qwen35-9b-gguf-q40 | not recordedextraction | 6.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc e5fa21270174a387 | |
| qwen35-9b-gguf-q4km | not recordedextraction | 23.6% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 71580a34709d8143 | |
| qwen3.8-27b-q4_0 | not recordedextraction | 69.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 0464331c99998f53 | |
| k2-36b-Q4_1 | not recordedextraction | 50.0% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc a1e412de2582e21b | |
| k2-36b-Q5_1 | not recordedextraction | 58.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 6e470ceac8f4401c | |
| k2-36b-Q5_0 | not recordedextraction | 56.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 53f943deaafde391 | |
| k2-36b-q4_0 | not recordedextraction | 65.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc c71d223f0ac921ec | |
| reap-60b-mlx-4bit-uniform | mlxextraction | 84.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 52349074a17dc8f5 | |
| reap-60b-20260910 | not recordedextraction | 47.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 067da0ca87821991 | |
| spark-x25-4b | not recordedextraction | 48.6% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 080b2b956a3c69aa | |
| qwen38-rebase-b10893 | not recordedextraction | 76.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 317df1bd5861fa04 | |
| k2-36b-q8-precision | not recordedextraction | 70.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 3a41a0a4d5f18851 | |
| k2-7b-bf16-precision | not recordedextraction | 37.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc d931535c0ebf24a6 | |
| k2-7b-q8-precision | not recordedextraction | 36.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc ed643e52a40b0e33 | |
| k2-7b-q4-precision | not recordedextraction | 37.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 577908cfcedc7d78 | |
| k2-4b-bf16-precision | not recordedextraction | 43.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 776fc6d33aa8cc34 | |
| k2-3.7b-q8-precision | not recordedextraction | 43.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 807203bd18193834 | |
| k2-3.7b-q4-precision | not recordedextraction | 38.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc b4a841d2cc241d1f | |
| gemma4-12b-midrange | not recordedextraction | 87.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc c10333ca24baf0ea | |
| lfm25-8b-a1b-midrange | not recordedextraction | 27.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc b3d59631a0a0785e | |
| qwen3-8b-midrange | not recordedextraction | 34.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 5fdc1502f911a148 | |
| qwen38-gsqrco-iq3s | not recordedextraction | 62.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 62838af8410253c2 | |
| bonsai-27b-ternary | not recordedextraction | 27.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 204142d8636f4fc2 | |
| qwen38-27b-np1 | not recordedextraction | 76.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 458eb55eee2cc3a3 | |
| k2-36b-vendor-k2b | not recordedvendor-k2b | 57.7% | Coverage 98.6% coverage | g d7448e0f5fc3f9afd 3b91280bb40372efc 62630cca2615a18f | |
| k2-36b-vendor-k2 | not recordedvendor-k2 | 48.6% | Coverage 97.2% coverage | g d7448e0f5fc3f9afd 3b91280bb40372efc 80a8857ae078d242 | |
| k2-36b-extraction | not recordedextraction | 54.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 5e938f4fe9784ee9 | |
| k2-36b-ctx32k | not recordedextraction | 54.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 345de57f0f3a686f | |
| k2-horizon-3.7b | not recordedextraction | 43.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc c1cd134c67de1621 | |
| k2-horizon-7b | not recordedextraction | 34.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 54f9df9571bc16e8 | |
| flashnext-4.27bpw-m64 | not recordedextraction | 58.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc a1f73dd74c73dbc6 | |
| qwen3-next-80b-reap | not recordedextraction | 97.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc cc55008d40e60573 | |
| glm-4.7-flash-mlx-q4 | mlxextraction | 40.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc b669c85bfa68b0b5 | |
| glm-4.7-flash-mlx-q8 | mlxextraction | 59.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 97ef09a912feb8be | |
| qwen3-8b-mlx-8bit | mlxextraction | 56.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc b9aaca6eee78b218 | |
| nemotron-lightning-30b-mlx-4bit | mlxextraction | 9.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 023ab1ea82fce0de | |
| granite-4.2-30b-gguf | llama.cppextraction | 90.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc ad5177ffa2d55fa5 | |
| ornith-1.5-35b-mlx-4bit | mlxextraction | 62.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 1fb1a5ad2ee784e3 | |
| granite-4.2-30b-mlx-q4 | mlxextraction | 100.0% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 48b3d526e6d6e6a3 | |
| qwen3-next-80b-mlx-4bit | mlxextraction | 95.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc d1b64a4d48cff9ef | |
| qwen38-flash-next | llama.cppextraction | 50.0% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc d248e89da830ce31 | |
| nemotron-lightning-30b | llama.cppextraction | 15.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 8f3225e9b30dcc6b | |
| qwen3-next-80b | llama.cppextraction | 97.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 5913b00d3e8c6b3d | |
| lfm2.5-8b-a1b | llama.cppextraction | 27.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 00612fdd4a39faf4 | |
| qwen3-next-80b-coder | llama.cppextraction | 56.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 74810b0cff4603b3 | |
| qwen3.6-35b | llama.cppextraction | 61.1% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 7a652a6a84f42ef5 | |
| qwen3-next-80b-thinking | llama.cppextraction | 58.3% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 8f8aa14bba9f91c3 | |
| qwen3-next-80b | llama.cppextraction | 97.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 5913b00d3e8c6b3d | |
| qwen3-next-80b-a3b-iq4xs | llama.cppextraction | 97.2% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 860f8678b21dfb1f | |
| glm-4.7-flash-q8 | llama.cppextraction | 48.6% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 3d836e2b73f9cd77 | |
| glm-4.7-flash-q8 | llama.cppextraction | n/a | Coverage 0% coverage | g d7448e0f5fc3f9afd 3b91280bb40372efc 3d836e2b73f9cd77 | |
| qwen3.5-122b-a10b-iq3 | llama.cppextraction | 45.8% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 5e7cd3243c74ce98 | |
| agents-a1-35b-b10618 | llama.cppextraction | 38.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 9a9ee4f1853f385a | |
| qwen3-coder-30b-b10618 | llama.cppextraction | 37.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc ffb75533d79bc27d | |
| qwen3.5-9b-q8-b10618 | llama.cppextraction | 30.6% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 989a2b1d084c0c53 | |
| gemma4-12b-b10618 | llama.cppextraction | 88.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 2c03d6ebad98fccc | |
| gemma4-e2b-b10618 | llama.cppextraction | 12.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 38b4b96050d74d17 | |
| gemma4-e4b-b10618 | llama.cppextraction | 13.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 84794dc91d0ce3b4 | |
| ornith-1.5-35b-q6-b10618 | llama.cppextraction | 34.7% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 355fd8b860ca54b1 | |
| devstral-small-2-24b-b10618 | llama.cppextraction | 62.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 836a74186efb247a | |
| gemma4-12b | llama.cppextraction | 88.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc ec3358e6aa5e1088 | |
| gemma4-e4b | llama.cppextraction | 13.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc addbcab5facddd0b | |
| gemma4-e2b | llama.cppextraction | 12.5% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc e268490afdec64ce |
Findings20 that name this suite
- 2026-09-18A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output
- 2026-09-18PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free
- 2026-09-18Every seat emits valid typed edges; three of five turn a hedged future statement into an asserted fact
- 2026-09-15IQ2_XXS took 27 points off world knowledge and 17 off tool invocation on identical weights
- 2026-09-15Abstention ranges 18-61 percent across the fleet, and declining is not the same skill as noticing a correctionrevised
- 2026-09-14Q4_0 decodes up to 2x faster than Q4_K_M at the same bit-width, and emits valid JSON more reliablyrevised
- 2026-09-13A 23% smaller IQ3_S build decodes 27% slower than Q4_0 on the same weights - the codebook costs more than the bytes it saves
- 2026-09-11The GGUF format result does not transfer to MLX -- symmetric quantisation is the WORST there
- 2026-09-11Runtime changes HOW a model phrases a refusal, and nullschema scores that as fabrication
- 2026-09-11Quantisation format changes output CONVENTIONS, not abstention judgementrevised
- 2026-09-11Badly-chosen mixed precision costs abstention; sensitivity-aware mixing protects itrevised
- 2026-09-10Quantisation damage is proportional to the capability present
- 2026-09-10Q8_0 and BF16 are behaviourally identical; full precision buys nothing
- 2026-09-10Detecting emptiness and rejecting a plausible candidate are different capabilities
- 2026-09-10The 6.7 GB tagger is the most epistemically reliable model in the fleet
- 2026-09-08mlx_lm silently ignores response_format
- 2026-09-08Schema enforcement is per-model, not per-runtime
- 2026-09-08The extraction suites can no longer separate the leading models
- 2026-09-08Expert pruning cost this model nothing measurable
- 2026-08-27Reasoning increases misattribution under authority pressure