Suite · contract← Suites
Following a convention it has to go and get
Will a model follow an output convention it can only obtain by asking for it?
- Cases
- 48
- Scored runs
- 25
- Models
- 25
- Top score
- 75.0%
SpecificationMethod · grading · caveat
- Method
- Four arms over the same extraction task, where the answer is always absent from the document. In silent, nothing is said. In told, the convention is stated in the system prompt. In fetch, a tool is offered that returns the convention and the prompt points at it. In unmentioned, the same tool is present but nothing refers to it. The convention is an arbitrary sentinel string, so it cannot be recalled from training.
- Grading
- Binary. A case passes only when the field carries the exact sentinel. Abstaining in the model's own words fails, and is classified separately from inventing a value.
- Caveat
- A grader bug conflated a missing field with a present null field until 2026-09-09, mislabelling every null abstention as unparseable output. Pass rates were unaffected but failure-mode breakdowns before that date are wrong.
- Version
contract/2- Cases
- 48
Recorded runs25 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite bonsai-27b-mlx2-cortext 75.0% 95% interval 61.2–85.1
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| bonsai-27b-mlx2-cortext | 75.0% | 61.2–85.1 | Passed 36/48 | mlxextraction | 2026-09-14 |
| bonsai-27b-mlx2-m4pro | 75.0% | 61.2–85.1overlaps the top interval | Passed 36/48 | mlxextraction | 2026-09-14 |
| qwen3.8-27b-mlx-4bit | 75.0% | 61.2–85.1overlaps the top interval | Passed 36/48 | mlxextraction | 2026-09-08 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 75.0% | 61.2–85.1overlaps the top interval | Passed 36/48 | llama.cppextraction | 2026-09-14 |
| BTL-4 (Q4_K_M)btl-4-q4km | 70.8% | 56.8–81.8overlaps the top interval | Passed 34/48 | llama.cppextraction | 2026-09-15 |
| minicpm5-2b | 70.8% | 56.8–81.8overlaps the top interval | Passed 34/48 | llama.cppextraction | 2026-09-15 |
| ornith-1.5-9b | 68.8% | 54.7–80.1overlaps the top interval | Passed 33/48 | llama.cppextraction | 2026-09-12 |
| Gemma 4 12Bgemma4-12b | 66.7% | 50.3–79.8overlaps the top interval | Passed 24/36 | llama.cppextraction | 2026-09-08 |
| granite-4.2-30b | 66.7% | 50.3–79.8overlaps the top interval | Passed 24/36 | llama.cppextraction | 2026-09-08 |
| ornith-35b-a3b-q40 | 66.7% | 52.5–78.3overlaps the top interval | Passed 32/48 | llama.cppextraction | 2026-09-12 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 64.6% | 50.4–76.6overlaps the top interval | Passed 31/48 | llama.cppextraction | 2026-09-15 |
| qwen3-next-80b-reap-mlx-mixed36 | 54.2% | 40.3–67.4overlaps the top interval | Passed 26/48 | mlxextraction | 2026-09-08 |
| qwen3-next-80b-reap-mlx-q4 | 47.2% | 32.0–63.0overlaps the top interval | Passed 17/36 | mlxextraction | 2026-09-08 |
| mlx-community-Qwen3.6-35B-A3B-4bit | 27.1% | 16.6–41.0 | Passed 13/48 | mlxextraction | 2026-09-08 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 25.0% | 14.9–38.8 | Passed 12/48 | llama.cppextraction | 2026-09-15 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 25.0% | 14.9–38.8 | Passed 12/48 | llama.cppextraction | 2026-09-14 |
| k2-horizon-7b | 22.9% | 13.3–36.5 | Passed 11/48 | llama.cppextraction | 2026-09-09 |
| ternary-bonsai-8b | 22.9% | 13.3–36.5 | Passed 11/48 | llama.cppextraction | 2026-09-17 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 20.8% | 11.7–34.3 | Passed 10/48 | llama.cppextraction | 2026-09-14 |
| flashnext-4.27bpw-m64 | 0.0% | 0.0–7.4 | Passed 0/48 | llama.cppextraction | — |
| q36-toolparser | 0.0% | 0.0–7.4 | Passed 0/48 | llama.cppextraction | — |
| Qwen3-Next-80B-A3B Instruct, REAP prunedqwen3-next-80b-reap | 0.0% | 0.0–7.4 | Passed 0/48 | llama.cppextraction | — |
| reapmix36-rapid-mlx | 0.0% | 0.0–9.6 | Passed 0/36 | mlxextraction | — |
| reapq4-mlx_lm | 0.0% | 0.0–9.6 | Passed 0/36 | mlxextraction | — |
| reapq4-rapid-mlx | 0.0% | 0.0–9.6 | Passed 0/36 | mlxextraction | — |
Run log25 logged runs of this suite
Every logged run, newest first (25)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| ternary-bonsai-8b | not recordedextraction | 22.9% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c b5bcdf7e22a61983 | |
| minicpm5-2b | not recordedextraction | 70.8% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 09ef74ce83b54a17 | |
| btl-4-q4km | not recordedextraction | 70.8% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c c5f63e89ad62fa86 | |
| btl-4 | not recordedextraction | 64.6% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c bf1f8546a0e1b97c | |
| bonsai-27b-1bit | not recordedextraction | 25.0% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c f3e9344f382b675d | |
| bonsai-27b-mlx2-cortext | mlxextraction | 75.0% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 1db1392d6a3fe16b | |
| bonsai-8b-1bit | not recordedextraction | 20.8% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c a575849c8bd49029 | |
| bonsai-27b-mlx2-m4pro | mlxextraction | 75.0% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 5836be70b11516a5 | |
| ternary-bonsai-27b | not recordedextraction | 75.0% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c e5777a35ca1e5f10 | |
| qwen3.8-9b-distill | not recordedextraction | 25.0% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 5a052624057c52ff | |
| ornith-1.5-9b | not recordedextraction | 68.8% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 1bf634506e78523a | |
| ornith-35b-a3b-q40 | not recordedextraction | 66.7% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 16c1587da02e495a | |
| k2-horizon-7b | not recordedextraction | 22.9% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c e127d0663cbe7e98 | |
| mlx-community-Qwen3.6-35B-A3B-4bit | mlxextraction | 27.1% | Coverage 48/48 | g 486601ba23034dd6d 1fe53d30cb5480f6c 4fc6c3fdb9c9b02c | |
| q36-toolparser | not recordedextraction | n/a | Coverage 0% coverage | g 486601ba23034dd6d 1fe53d30cb5480f6c 2225a59526198b68 | |
| qwen3.8-27b-mlx-4bit | mlxextraction | 75.0% | Coverage 48/48 | g 486601ba23034dd6d 1fe53d30cb5480f6c 2750a525d9018a88 | |
| flashnext-4.27bpw-m64 | not recordedextraction | n/a | Coverage 0% coverage | g 486601ba23034dd6d 1fe53d30cb5480f6c 92b28699e44d08e7 | |
| qwen3-next-80b-reap | not recordedextraction | n/a | Coverage 0% coverage | g 486601ba23034dd6d 1fe53d30cb5480f6c cbb517440b51fc70 | |
| qwen3-next-80b-reap-mlx-mixed36 | mlxextraction | 54.2% | Coverage 48/48 | g 486601ba23034dd6d 1fe53d30cb5480f6c e059879c3195b24c | |
| qwen3-next-80b-reap-mlx-q4 | mlxextraction | 47.2% | Coverage 36/36 | g 2dd146852463f032d ab4bb33923bd0ed6c 14fac1620eb065e0 | |
| reapq4-mlx_lm | mlxextraction | n/a | Coverage 0% coverage | g 2dd146852463f032d ab4bb33923bd0ed6c 224bdfd4a5d2e672 | |
| reapmix36-rapid-mlx | mlxextraction | n/a | Coverage 0% coverage | g 2dd146852463f032d ab4bb33923bd0ed6c 46d4a279b80fca4d | |
| reapq4-rapid-mlx | mlxextraction | n/a | Coverage 0% coverage | g 2dd146852463f032d ab4bb33923bd0ed6c 788a11f836a3345b | |
| granite-4.2-30b | not recordedextraction | 66.7% | Coverage 36/36 | g 2dd146852463f032d ab4bb33923bd0ed6c cc8ca71e6e299aee | |
| gemma4-12b | not recordedextraction | 66.7% | Coverage 36/36 | g 2dd146852463f032d ab4bb33923bd0ed6c 0d1de531e8b00c72 |