Suite · grounding← Suites
Grounding and output mode
Does the mitigation work, and what does each output mode cost?
- Cases
- 125
- Scored runs
- 61
- Models
- 60
- Top score
- 100.0%
SpecificationMethod · grading · caveat
- Method
- The same question is asked of the same document under different output modes, including free text and constrained decoding. Documents carry line addresses so any claim can be traced back to a source line.
- Grading
- A case passes when the answer is present in the document and correctly attributed. Answers taken from the wrong line count as misattribution, not as fabrication.
- Caveat
- Requires a server that honours response_format. Results from mlx_lm are not valid because it ignores the field.
- Version
grounding/1- Cases
- 125
Recorded runs61 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite Qwen3-Next-80B-A3B Thinking 100.0% 95% interval 20.7–100.0
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| Qwen3-Next-80B-A3B Thinkingqwen3-next-80b-thinking | 100.0% | 20.7–100.0 | Passed 1/1 | llama.cppstudy-default | 2026-08-27 |
| qwen3.8-27b-mlx-4bitunenforced | 92.0% | 85.9–95.6overlaps the top interval | Passed 115/125 | mlxextraction | 2026-09-07 |
| edge0-35b | 89.6% | 83.0–93.8overlaps the top interval | Passed 112/125 | llama.cppextraction | 2026-09-16 |
| Qwen3-Next-80B-A3B Instructqwen3-next-80b | 88.8% | 82.1–93.2overlaps the top interval | Passed 111/125 | llama.cppextraction | 2026-09-05 |
| qwen3-next-80b-a3b-iq4xs | 88.8% | 82.1–93.2overlaps the top interval | Passed 111/125 | llama.cppextraction | 2026-08-26 |
| qwen3-next-80b-mlx-4bitunenforced | 87.2% | 80.2–92.0overlaps the top interval | Passed 109/125 | mlxextraction | 2026-09-07 |
| Qwen3-Next-80B-A3B Instruct, REAP prunedqwen3-next-80b-reap | 87.2% | 80.2–92.0overlaps the top interval | Passed 109/125 | llama.cppextraction | 2026-09-08 |
| granite-4.2-30b-mlx-q4unenforced | 86.4% | 79.3–91.3overlaps the top interval | Passed 108/125 | mlxextraction | 2026-09-08 |
| Muse-Glimmer 30Bmuse-glimmer-30b | 85.6% | 78.4–90.7overlaps the top interval | Passed 107/125 | llama.cppextraction | 2026-09-08 |
| Gemma 4 12Bgemma4-12b | 84.0% | 76.6–89.4overlaps the top interval | Passed 105/125 | llama.cppextraction | 2026-08-24 |
| gemma4-12b-b10618 | 84.0% | 76.6–89.4overlaps the top interval | Passed 105/125 | llama.cppextraction | 2026-08-25 |
| Granite 4.2 30Bgranite-4.2-30b-gguf | 84.0% | 76.6–89.4overlaps the top interval | Passed 105/125 | llama.cppextraction | 2026-09-08 |
| Qwen3.6 35B-A3Bqwen3.6-35b | 82.4% | 74.8–88.1overlaps the top interval | Passed 103/125 | llama.cppextraction | 2026-08-27 |
| flashnext-4.27bpw-m64 | 81.6% | 73.9–87.4overlaps the top interval | Passed 102/125 | llama.cppextraction | 2026-09-08 |
| ornith-1.5-9b | 81.6% | 73.9–87.4overlaps the top interval | Passed 102/125 | llama.cppextraction | 2026-09-12 |
| Qwen3.5 122B-A10Bqwen3.5-122b-a10b-iq3 | 80.8% | 73.0–86.7overlaps the top interval | Passed 101/125 | llama.cppextraction | 2026-08-26 |
| ornith-9b-kvq4 | 80.0% | 72.1–86.1overlaps the top interval | Passed 100/125 | llama.cppextraction | 2026-09-12 |
| Qwen3.8 27Bqwen3.8-27b | 80.0% | 72.1–86.1overlaps the top interval | Passed 100/125 | llama.cppextraction | 2026-08-24 |
| qwen3.8-27b-noreason | 80.0% | 72.1–86.1overlaps the top interval | Passed 100/125 | llama.cppextraction | 2026-09-07 |
| Devstral Small 2 24Bdevstral-small-2-24b-b10618 | 79.2% | 71.3–85.4overlaps the top interval | Passed 99/125 | llama.cppextraction | 2026-08-25 |
| Qwen3.8-Flash-Nextqwen38-flash-next | 79.2% | 71.3–85.4overlaps the top interval | Passed 99/125 | llama.cppextraction | 2026-09-05 |
| Qwen3 Coder 30Bqwen3-coder-30b | 77.6% | 69.5–84.0overlaps the top interval | Passed 97/125 | llama.cppextraction | 2026-08-24 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 76.8% | 68.7–83.3overlaps the top interval | Passed 96/125 | llama.cppextraction | 2026-09-15 |
| BTL-4 (Q4_K_M)btl-4-q4km | 76.8% | 68.7–83.3overlaps the top interval | Passed 96/125 | llama.cppextraction | 2026-09-15 |
| glm-4.7-flash-mlx-q8unenforced | 76.8% | 68.7–83.3overlaps the top interval | Passed 96/125 | mlxextraction | 2026-09-08 |
| qwen3-coder-30b-b10618 | 76.8% | 68.7–83.3overlaps the top interval | Passed 96/125 | llama.cppextraction | 2026-08-26 |
| qwen3-next-80b-coder | 76.8% | 68.7–83.3overlaps the top interval | Passed 96/125 | llama.cppextraction | 2026-08-27 |
| GLM-4.7-Flashglm-4.7-flash-q8 | 76.0% | 67.8–82.6overlaps the top interval | Passed 95/125 | llama.cppextraction | 2026-08-26 |
| ornith-35b-a3b-q40 | 76.0% | 67.8–82.6overlaps the top interval | Passed 95/125 | llama.cppextraction | 2026-09-12 |
| qwen3-8b-mlx-8bitunenforced | 74.4% | 66.1–81.2overlaps the top interval | Passed 93/125 | mlxextraction | 2026-09-08 |
| qwen3.5-4b-base | 74.4% | 66.1–81.2overlaps the top interval | Passed 93/125 | llama.cppextraction | 2026-09-13 |
| Ornith 1.5 35Bornith-1.5-35b-q6-b10618 | 72.8% | 64.4–79.8overlaps the top interval | Passed 91/125 | llama.cppextraction | 2026-08-25 |
| qwen3.5-9b-q8 | 72.0% | 63.6–79.1overlaps the top interval | Passed 90/125 | llama.cppextraction | 2026-08-24 |
| qwen3.5-9b-q8-b10618 | 72.0% | 63.6–79.1overlaps the top interval | Passed 90/125 | llama.cppextraction | 2026-08-25 |
| Agents-A1 35Bagents-a1-35b | 71.2% | 62.7–78.4overlaps the top interval | Passed 89/125 | llama.cppextraction | 2026-08-24 |
| agents-a1-35b-b10618 | 71.2% | 62.7–78.4overlaps the top interval | Passed 89/125 | llama.cppextraction | 2026-08-26 |
| bonsai-27b-mlx2-cortextunenforced | 71.2% | 62.7–78.4overlaps the top interval | Passed 89/125 | mlxextraction | 2026-09-14 |
| bonsai-27b-mlx2-m4prounenforced | 71.2% | 62.7–78.4overlaps the top interval | Passed 89/125 | mlxextraction | 2026-09-14 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 68.8% | 60.2–76.3overlaps the top interval | Passed 86/125 | llama.cppextraction | 2026-09-14 |
| qwen3-next-80b-a3b-iq4xs-smoke | 68.0% | 48.4–82.8overlaps the top interval | Passed 17/25 | llama.cppextraction | — |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 67.2% | 58.6–74.8overlaps the top interval | Passed 84/125 | llama.cppextraction | 2026-09-14 |
| ornith-1.5-35b-mlx-4bitunenforced | 67.2% | 58.6–74.8overlaps the top interval | Passed 84/125 | mlxextraction | 2026-09-08 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 67.2% | 58.6–74.8overlaps the top interval | Passed 84/125 | llama.cppextraction | 2026-09-14 |
| bonsai2-27b-pq2 | 66.4% | 57.7–74.1overlaps the top interval | Passed 83/125 | llama.cppextraction | 2026-09-18 |
| bonsai2-27b-ptq1 | 66.4% | 57.7–74.1overlaps the top interval | Passed 83/125 | llama.cppextraction | 2026-09-18 |
| Qwen3-Next-80B-A3B Thinkingqwen3-next-80b-thinking | 65.6% | 56.9–73.4overlaps the top interval | Passed 82/125 | llama.cppextraction | 2026-08-27 |
| neohorse-1-4b | 63.2% | 54.5–71.1overlaps the top interval | Passed 79/125 | llama.cppextraction | 2026-09-13 |
| gemma4-e4b | 62.4% | 53.7–70.4overlaps the top interval | Passed 78/125 | llama.cppextraction | 2026-08-24 |
| gemma4-e4b-b10618 | 62.4% | 53.7–70.4overlaps the top interval | Passed 78/125 | llama.cppextraction | 2026-08-25 |
| minicpm5-2b | 62.4% | 53.7–70.4overlaps the top interval | Passed 78/125 | llama.cppextraction | 2026-09-15 |
| Gemma 4 E2Bgemma4-e2b | 61.6% | 52.8–69.7overlaps the top interval | Passed 77/125 | llama.cppextraction | 2026-08-24 |
| gemma4-e2b-b10618 | 61.6% | 52.8–69.7overlaps the top interval | Passed 77/125 | llama.cppextraction | 2026-08-25 |
| Nemotron 3.5 Lightning 30Bnemotron-lightning-30b | 60.0% | 51.2–68.2overlaps the top interval | Passed 75/125 | llama.cppextraction | 2026-09-05 |
| nemotron-lightning-30b-mlx-4bitunenforced | 56.8% | 48.0–65.2overlaps the top interval | Passed 71/125 | mlxextraction | 2026-09-08 |
| minicpm5-2b-q8-np1 | 56.0% | 47.2–64.4overlaps the top interval | Passed 70/125 | llama.cppextraction | 2026-09-17 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 55.2% | 46.5–63.6overlaps the top interval | Passed 69/125 | llama.cppextraction | 2026-09-14 |
| k2-horizon-7b | 55.2% | 46.5–63.6overlaps the top interval | Passed 69/125 | llama.cppextraction | 2026-09-09 |
| glm-4.7-flash-mlx-q4unenforced | 53.6% | 44.9–62.1overlaps the top interval | Passed 67/125 | mlxextraction | 2026-09-08 |
| ternary-bonsai-8b | 51.2% | 42.5–59.8overlaps the top interval | Passed 64/125 | llama.cppextraction | 2026-09-17 |
| minicpm5-2b-exl3 | 48.0% | 39.4–56.7overlaps the top interval | Passed 60/125 | llama.cppextraction | — |
| LFM 2.5 8B-A1Blfm2.5-8b-a1b | 47.2% | 38.7–55.9overlaps the top interval | Passed 59/125 | llama.cppextraction | 2026-08-31 |
Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.
Run log63 logged runs of this suite
Every logged run, newest first (63)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| bonsai2-27b-ptq1 | not recordedextraction | 66.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 0407c0f9d876e4a3 | |
| bonsai2-27b-pq2 | not recordedextraction | 66.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 1685634443e1889f | |
| minicpm5-2b-q8-np1 | not recordedextraction | 56.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 5ac3f164bce7795b | |
| minicpm5-2b-exl3 | not recordedextraction | 60.0% | Coverage 80% coverage | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac deed1276a964eb4d | |
| ternary-bonsai-8b | not recordedextraction | 51.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac a964482c0a6fefb9 | |
| edge0-35b | not recordedextraction | 89.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac eb1910ec33534645 | |
| minicpm5-2b | not recordedextraction | 62.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 8f32a36bd07ac12d | |
| btl-4-q4km | not recordedextraction | 76.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac b6421e71dc1b07d5 | |
| btl-4 | not recordedextraction | 76.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 7d9f8293fd874d62 | |
| bonsai-27b-1bit | not recordedextraction | 67.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 0f1cdb00af0adb60 | |
| bonsai-8b-1bit | not recordedextraction | 55.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 1ac5570cbae78ecd | |
| bonsai-27b-mlx2-cortext | mlxextraction | 71.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 4f595a317a905809 | |
| bonsai-27b-mlx2-m4pro | mlxextraction | 71.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 86fb3d4c7b477c6c | |
| ternary-bonsai-27b | not recordedextraction | 68.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 3b0afe72fb0c0121 | |
| qwen3.8-9b-distill | not recordedextraction | 67.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 696fd8b30eed53f0 | |
| qwen3.5-4b-base | not recordedextraction | 74.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 0f7dbcd0b1d1afd4 | |
| neohorse-1-4b | not recordedextraction | 63.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac b382340e2e3ad6fa | |
| ornith-9b-kvq4 | not recordedextraction | 80.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 4c61ed44aed10e7a | |
| ornith-35b-a3b-q40 | not recordedextraction | 76.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 95c69b1144e96db8 | |
| ornith-1.5-9b | not recordedextraction | 81.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 4a6ad96362484d18 | |
| k2-horizon-7b | not recordedextraction | 55.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 30ddb902b06a0c00 | |
| flashnext-4.27bpw-m64 | not recordedextraction | 81.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 2867d56b04f8bbb4 | |
| qwen3-next-80b-reap | not recordedextraction | 87.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 7e1bbf58551d87ef | |
| muse-glimmer-30b | llama.cppextraction | 85.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac d9d25da365e04cc8 | |
| glm-4.7-flash-mlx-q4 | mlxextraction | 53.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 2432689cc6751eea | |
| glm-4.7-flash-mlx-q8 | mlxextraction | 76.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 6d0a345ab4ecfa54 | |
| qwen3-8b-mlx-8bit | mlxextraction | 74.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac e34a89daf5181d58 | |
| nemotron-lightning-30b-mlx-4bit | mlxextraction | 56.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 15a43d7b2024f68c | |
| granite-4.2-30b-gguf | llama.cppextraction | 84.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac e353f603986dcdcd | |
| ornith-1.5-35b-mlx-4bit | mlxextraction | 67.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac a438aff9707ec8a1 | |
| granite-4.2-30b-mlx-q4 | mlxextraction | 86.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac db0a9174aa469c68 | |
| qwen3-next-80b-mlx-4bit | mlxextraction | 87.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 0c539d39afff5bd1 | |
| qwen3.8-27b-noreason | llama.cppextraction | 80.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 4a3b4b7c53219329 | |
| qwen3.8-27b-mlx-4bit | mlxextraction | 92.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 42a3cf291ff302bb | |
| qwen38-flash-next | llama.cppextraction | 79.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 9148fcdc2bc14fe4 | |
| nemotron-lightning-30b | llama.cppextraction | 60.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac f734b41b3caabfdd | |
| qwen3-next-80b | llama.cppextraction | 88.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac f34a40087b318cc6 | |
| lfm2.5-8b-a1b | llama.cppextraction | 47.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 90be8dffe935ab41 | |
| qwen3-next-80b-coder | llama.cppextraction | 76.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 140e26c8be639be7 | |
| qwen3.6-35b | llama.cppextraction | 82.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 279f3a4bd1ff0c7e | |
| qwen3.6-35b | llama.cppextraction | 86.3% | Coverage 73/73 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 279f3a4bd1ff0c7e | |
| qwen3-next-80b-thinking | llama.cppstudy-default | 100.0% | Coverage 1/1 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac cb1f0b0a42b6ac48 | |
| qwen3-next-80b-thinking | llama.cppextraction | 65.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac e644b21297e6198b | |
| qwen3-next-80b | llama.cppextraction | 88.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac f34a40087b318cc6 | |
| qwen3-next-80b-a3b-iq4xs | llama.cppextraction | 88.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac e3b83bfa70c1a417 | |
| qwen3-next-80b-a3b-iq4xs-smoke | llama.cppextraction | 73.9% | Coverage 92% coverage | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac e6920f17163814d7 | |
| glm-4.7-flash-q8 | llama.cppextraction | 76.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac f7116f1eb6a7f001 | |
| qwen3.5-122b-a10b-iq3 | llama.cppextraction | 80.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac b3059734754cd76c | |
| agents-a1-35b-b10618 | llama.cppextraction | 71.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac b6d15c86cef9c0f6 | |
| qwen3-coder-30b-b10618 | llama.cppextraction | 76.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 85522d05c3671c16 | |
| qwen3.5-9b-q8-b10618 | llama.cppextraction | 72.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 9e54265c33ff0f82 | |
| gemma4-12b-b10618 | llama.cppextraction | 84.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac ba97a71cfa0468a7 | |
| gemma4-e2b-b10618 | llama.cppextraction | 61.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac b18ad353c84d839e | |
| gemma4-e4b-b10618 | llama.cppextraction | 62.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac dc6988d619b065bf | |
| ornith-1.5-35b-q6-b10618 | llama.cppextraction | 72.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac bc2f7777482c6fb6 | |
| devstral-small-2-24b-b10618 | llama.cppextraction | 79.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac ed588aa2abdb41ef | |
| qwen3-coder-30b | llama.cppextraction | 77.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac ade06bde17b71832 | |
| qwen3.8-27b | llama.cppextraction | 80.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 1d989a6b00eb212c | |
| agents-a1-35b | llama.cppextraction | 71.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 3cb35299a047efb8 | |
| qwen3.5-9b-q8 | llama.cppextraction | 72.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac bd21ebb86ee6f863 | |
| gemma4-12b | llama.cppextraction | 84.0% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 31241ed46ed4d6a5 | |
| gemma4-e4b | llama.cppextraction | 62.4% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 267624907124ed7e | |
| gemma4-e2b | llama.cppextraction | 61.6% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac dee67653e2de45c6 |