- Cases
- varies
- Scored runs
- 53
- Models
- 53
- Top score
- 97.5%
SpecificationMethod · grading · caveat
This suite has runs in the record but no entry in the published suite documentation, so there is no stated method, grading rule or caveat to show.
Recorded runs53 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite Gemma 4 12B 97.5% 95% interval 87.1–99.6
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| Gemma 4 12Bgemma4-12b | 97.5% | 87.1–99.6 | Passed 39/40 | llama.cppextraction | 2026-09-09 |
| gemma4-12b-midrange | 97.5% | 87.1–99.6overlaps the top interval | Passed 39/40 | llama.cppextraction | 2026-09-10 |
| qwen38-27b-q3kxl-chad | 92.5% | 80.1–97.4overlaps the top interval | Passed 37/40 | llama.cppextraction | 2026-09-18 |
| ornith-35b-a3b-q40 | 85.0% | 70.9–92.9overlaps the top interval | Passed 34/40 | llama.cppextraction | 2026-09-12 |
| ornith-35b-a3b-q40-recheck | 85.0% | 70.9–92.9overlaps the top interval | Passed 34/40 | llama.cppextraction | 2026-09-17 |
| bonsai2-27b-pq2 | 82.5% | 68.0–91.3overlaps the top interval | Passed 33/40 | llama.cppextraction | 2026-09-18 |
| bonsai2-27b-ptq1 | 82.5% | 68.0–91.3overlaps the top interval | Passed 33/40 | llama.cppextraction | 2026-09-18 |
| k2-36b-vendor-k2b | 82.5% | 68.0–91.3overlaps the top interval | Passed 33/40 | llama.cppvendor-k2b | 2026-09-10 |
| qwen3.8-27b-q4_0 | 80.0% | 65.2–89.5overlaps the top interval | Passed 32/40 | llama.cppextraction | 2026-09-11 |
| reap-60b-gguf-q4km | 80.0% | 65.2–89.5overlaps the top interval | Passed 32/40 | llama.cppextraction | 2026-09-10 |
| k2-36b-a4b-q4 | 78.9% | 56.7–91.5overlaps the top interval | Passed 15/19 | llama.cppextraction | 2026-09-09 |
| diffusiongemma-26b-mlx | 77.5% | 62.5–87.7overlaps the top interval | Passed 31/40 | mlxextraction | 2026-09-11 |
| k2-36b-vendor-k2 | 77.5% | 62.5–87.7overlaps the top interval | Passed 31/40 | llama.cppvendor-k2 | 2026-09-09 |
| qwen38-27b-np1 | 77.5% | 62.5–87.7overlaps the top interval | Passed 31/40 | llama.cppextraction | 2026-09-10 |
| qwen38-gsqrco-iq3s | 77.5% | 62.5–87.7overlaps the top interval | Passed 31/40 | llama.cppextraction | 2026-09-10 |
| qwen38-rebase-b10893 | 77.5% | 62.5–87.7overlaps the top interval | Passed 31/40 | llama.cppextraction | 2026-09-10 |
| qwen38-vendor-qwen-think | 77.5% | 62.5–87.7overlaps the top interval | Passed 31/40 | llama.cppvendor-qwen-think | 2026-09-10 |
| k2-36b-ctx128k-q8 | 65.0% | 49.5–77.9 | Passed 26/40 | llama.cppextraction | 2026-09-09 |
| k2-36b-ctx32k | 65.0% | 49.5–77.9 | Passed 26/40 | llama.cppextraction | 2026-09-09 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 62.5% | 47.0–75.8 | Passed 25/40 | llama.cppextraction | 2026-09-15 |
| ornith-1.5-9b | 62.5% | 47.0–75.8 | Passed 25/40 | llama.cppextraction | 2026-09-12 |
| k2-36b-q4_0 | 57.5% | 42.2–71.5 | Passed 23/40 | llama.cppextraction | 2026-09-11 |
| minicpm5-2b-q8-np1 | 57.5% | 42.2–71.5 | Passed 23/40 | llama.cppextraction | 2026-09-17 |
| BTL-4 (Q4_K_M)btl-4-q4km | 55.0% | 39.8–69.3 | Passed 22/40 | llama.cppextraction | 2026-09-15 |
| k2-36b-q8-precision | 55.0% | 39.8–69.3 | Passed 22/40 | llama.cppextraction | 2026-09-10 |
| edge0-35b | 52.5% | 37.5–67.1 | Passed 21/40 | llama.cppextraction | 2026-09-16 |
| k2-3.7b-q4-precision | 52.5% | 37.5–67.1 | Passed 21/40 | llama.cppextraction | 2026-09-10 |
| k2-36b-extraction | 52.5% | 37.5–67.1 | Passed 21/40 | llama.cppextraction | 2026-09-09 |
| k2-3.7b | 50.0% | 35.2–64.8 | Passed 20/40 | llama.cppextraction | 2026-09-09 |
| k2-3.7b-q8-precision | 50.0% | 35.2–64.8 | Passed 20/40 | llama.cppextraction | 2026-09-10 |
| k2-4b-bf16-precision | 50.0% | 35.2–64.8 | Passed 20/40 | llama.cppextraction | 2026-09-10 |
| reap-60b | 47.5% | 32.9–62.5 | Passed 19/40 | llama.cppextraction | 2026-09-09 |
| reap-60b-20260910 | 47.5% | 32.9–62.5 | Passed 19/40 | llama.cppextraction | 2026-09-10 |
| reap-60b-mlx-4bit-uniform | 47.5% | 32.9–62.5 | Passed 19/40 | mlxextraction | 2026-09-11 |
| qwen3-8b-midrange | 45.0% | 30.7–60.2 | Passed 18/40 | llama.cppextraction | 2026-09-10 |
| bonsai-27b-mlx2-cortext | 42.5% | 28.5–57.8 | Passed 17/40 | mlxextraction | 2026-09-14 |
| bonsai-27b-mlx2-m4pro | 42.5% | 28.5–57.8 | Passed 17/40 | mlxextraction | 2026-09-14 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 42.5% | 28.5–57.8 | Passed 17/40 | llama.cppextraction | 2026-09-14 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 40.0% | 26.3–55.4 | Passed 16/40 | llama.cppextraction | 2026-09-14 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 40.0% | 26.3–55.4 | Passed 16/40 | llama.cppextraction | 2026-09-14 |
| neohorse-1-4b | 40.0% | 26.3–55.4 | Passed 16/40 | llama.cppextraction | 2026-09-13 |
| qwen3.5-4b-base | 40.0% | 26.3–55.4 | Passed 16/40 | llama.cppextraction | 2026-09-13 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 40.0% | 26.3–55.4 | Passed 16/40 | llama.cppextraction | 2026-09-14 |
| glm-4.7-flash-q40 | 37.5% | 24.2–53.0 | Passed 15/40 | llama.cppextraction | 2026-09-12 |
| k2-7b-q4-precision | 37.5% | 24.2–53.0 | Passed 15/40 | llama.cppextraction | 2026-09-10 |
| minicpm5-2b | 37.5% | 24.2–53.0 | Passed 15/40 | llama.cppextraction | 2026-09-15 |
| minicpm5-2b-exl3 | 37.5% | 24.2–53.0 | Passed 15/40 | llama.cppextraction | 2026-09-17 |
| ternary-bonsai-8b | 37.5% | 24.2–53.0 | Passed 15/40 | llama.cppextraction | 2026-09-17 |
| k2-7b | 30.0% | 18.1–45.4 | Passed 12/40 | llama.cppextraction | 2026-09-09 |
| k2-7b-bf16-precision | 30.0% | 18.1–45.4 | Passed 12/40 | llama.cppextraction | 2026-09-10 |
| k2-7b-q8-precision | 30.0% | 18.1–45.4 | Passed 12/40 | llama.cppextraction | 2026-09-10 |
| lfm25-8b-a1b-midrange | 30.0% | 18.1–45.4 | Passed 12/40 | llama.cppextraction | 2026-09-10 |
| spark-x25-4b | 20.0% | 10.5–34.8 | Passed 8/40 | llama.cppextraction | 2026-09-10 |
Run log56 logged runs of this suite
Every logged run, newest first (56)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| bonsai2-27b-ptq1 | not recordedextraction | 82.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 40c68eea7655c7e1 | |
| qwen38-27b-q3kxl-chad | not recordedextraction | 92.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 7f85527901732b16 | |
| bonsai2-27b-ptq1 | not recordedextraction | 100.0% | Coverage 9/9 | g 9acfba1b704feb28d e97835de9e6fc227c 40c68eea7655c7e1 | |
| bonsai2-27b-pq2 | not recordedextraction | 82.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 61b04f512aeac639 | |
| ornith-35b-a3b-q40-recheck | not recordedextraction | 85.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 2124361885388e32 | |
| minicpm5-2b-q8-np1 | not recordedextraction | 57.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c ff823de24a74e591 | |
| minicpm5-2b-exl3 | not recordedextraction | 37.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c f0bd4498f4e1a804 | |
| ternary-bonsai-8b | not recordedextraction | 37.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 0b08940f0421efe8 | |
| minicpm5-2b-exl3 | not recordedextraction | 80.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c d413710f2e637004 | |
| edge0-35b | not recordedextraction | 52.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c c4a49113da942eeb | |
| minicpm5-2b | not recordedextraction | 37.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c d8fc1797ed992595 | |
| btl-4-q4km | not recordedextraction | 55.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 7cf929c72ed89ccb | |
| btl-4 | not recordedextraction | 62.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 0104404569f7c535 | |
| bonsai-27b-1bit | not recordedextraction | 40.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c c66967b2cb84959c | |
| bonsai-8b-1bit | not recordedextraction | 40.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 1f1610844ef40acb | |
| bonsai-27b-mlx2-cortext | mlxextraction | 42.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 88e5e9056622def0 | |
| bonsai-27b-mlx2-m4pro | mlxextraction | 42.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c a4f61e2b7e694288 | |
| ternary-bonsai-27b | not recordedextraction | 42.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 3a4970c1d0eed32d | |
| qwen3.8-9b-distill | not recordedextraction | 40.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 6a67c959e17c98f5 | |
| qwen3.5-4b-base | not recordedextraction | 40.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 314d1643f96e91a9 | |
| neohorse-1-4b | not recordedextraction | 40.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 0ca5e9b594862393 | |
| glm-4.7-flash-q40 | not recordedextraction | 37.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 9d24f8d5c84a9e2e | |
| ornith-35b-a3b-q40 | not recordedextraction | 85.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 36403d17c401159a | |
| ornith-1.5-9b | not recordedextraction | 62.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c ce57d773fcc0cbcd | |
| qwen3.8-27b-q4_0 | not recordedextraction | 80.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 4fb569ecc91140c9 | |
| k2-36b-q4_0 | not recordedextraction | 57.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c d2167f66564312b7 | |
| reap-60b-mlx-4bit-uniform | mlxextraction | 47.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 76804a68d58c5eb6 | |
| diffusiongemma-26b-mlx | mlxextraction | 77.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c b0b19b278db1058f | |
| reap-60b-gguf-q4km | not recordedextraction | 80.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 361b96fbc9be289c | |
| reap-60b-20260910 | not recordedextraction | 47.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c a9d79c1de500e4b4 | |
| spark-x25-4b | not recordedextraction | 20.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 7eb3a5699dc38e49 | |
| qwen38-rebase-b10893 | not recordedextraction | 77.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 90bd067cd59ed0c2 | |
| k2-36b-q8-precision | not recordedextraction | 55.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c c95b13073128b76a | |
| k2-7b-bf16-precision | not recordedextraction | 30.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 1705a034b7c84210 | |
| k2-7b-q8-precision | not recordedextraction | 30.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c e20e4b1b964fa50d | |
| k2-7b-q4-precision | not recordedextraction | 37.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 3bbb4039e7a64a8b | |
| k2-4b-bf16-precision | not recordedextraction | 50.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 770c2ce3d2f16e96 | |
| k2-3.7b-q8-precision | not recordedextraction | 50.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c f654ce7b2d1ac877 | |
| k2-3.7b-q4-precision | not recordedextraction | 52.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 324471b050d0453b | |
| gemma4-12b-midrange | not recordedextraction | 97.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 4bbaa37911174c88 | |
| lfm25-8b-a1b-midrange | not recordedextraction | 30.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 717aaa5a6a785c10 | |
| qwen3-8b-midrange | not recordedextraction | 45.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 39d8f0df5c0c0e6f | |
| qwen38-vendor-qwen-think | not recordedvendor-qwen-think | 77.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c e78fad9e97d45282 | |
| qwen38-gsqrco-iq3s | not recordedextraction | 77.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 624bd85753786f4e | |
| bonsai-27b-ternary | not recordedextraction | 42.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 121caa6718989cf9 | |
| qwen38-27b-np1 | not recordedextraction | 77.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c cf60f4eb1ae532f8 | |
| k2-36b-vendor-k2b | not recordedvendor-k2b | 82.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c d16f8472c1b1d87d | |
| k2-36b-vendor-k2 | not recordedvendor-k2 | 77.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c e6f5010ab18f79e7 | |
| k2-36b-extraction | not recordedextraction | 52.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 9cc504b6f261a5b3 | |
| k2-36b-ctx128k-q8 | not recordedextraction | 65.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 082b24620003bf78 | |
| k2-36b-ctx32k | not recordedextraction | 65.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 082b24620003bf78 | |
| k2-36b-a4b-q4 | not recordedextraction | 78.9% | Coverage 19/19 | g 9acfba1b704feb28d e97835de9e6fc227c 082b24620003bf78 | |
| k2-7b | not recordedextraction | 30.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c e229d76d7d696757 | |
| k2-3.7b | not recordedextraction | 50.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 13dc41ad1b2821d9 | |
| gemma4-12b | not recordedextraction | 97.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 585e7383f726d519 | |
| reap-60b | not recordedextraction | 47.5% | Coverage 40/40 | g d511b635ce7bbfd3d d7ae9cbb38fafbe9c f6b9a462ff423a87 |
Findings11 that name this suite
- 2026-09-18A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output
- 2026-09-18PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free
- 2026-09-18edge0 scores 0/8 on worldview's conflicting arm and 81% on a probe that varies one thing at a time; the arm was measuring three skills at once
- 2026-09-17EXL3/QTIP at 4 bpw costs nothing measurable in capability and three quarters of the throughput
- 2026-09-17One baseline was 25 points off and nothing in the stamp could explain it; the other six reproduced exactly
- 2026-09-15IQ2_XXS took 27 points off world knowledge and 17 off tool invocation on identical weights
- 2026-09-15Abstention ranges 18-61 percent across the fleet, and declining is not the same skill as noticing a correctionrevised
- 2026-09-15MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites
- 2026-09-13A 23% smaller IQ3_S build decodes 27% slower than Q4_0 on the same weights - the codebook costs more than the bytes it saves
- 2026-09-10Q8_0 and BF16 are behaviourally identical; full precision buys nothing
- 2026-09-10The 6.7 GB tagger is the most epistemically reliable model in the fleet