Suite · synthesis← Suites
Turning tool results into an answer
Can a model synthesise a set of tool results into a usable, attributed answer?
- Cases
- 60
- Scored runs
- 44
- Models
- 44
- Top score
- 100.0%
SpecificationMethod · grading · caveat
- Method
- A question plus results, each carrying a REAL address: a turn id, an OKF doc slug, or an fcbench: case reference. Six arms vary only the result set - sufficient, distractor, conflicting, insufficient, stale, and single_source (one external document, no corroboration).
- Grading
- Mechanical, no judge model. The answer must carry the right value AND cite a valid address in the real grammar; an invented address fails on the grammar rather than on opinion. Conflicts must be flagged with both addresses, absent answers declared, and single-source claims qualified - neither asserted bare nor refused.
- Caveat
- Every model tested scores 0/10 on single_source, all overclaiming, despite an explicit instruction. That arm currently measures a capability nothing has rather than separating models.
- Version
synthesis/2- Cases
- 60
Recorded runs44 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite qwen3-next-80b-reap-mlx-mixed36 100.0% 95% interval 94.0–100.0
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| qwen3-next-80b-reap-mlx-mixed36 | 100.0% | 94.0–100.0 | Passed 60/60 | mlxextraction | 2026-09-09 |
| k2-36b-vendor-k2 | 93.3% | 84.1–97.4overlaps the top interval | Passed 56/60 | llama.cppvendor-k2 | 2026-09-10 |
| k2-36b-vendor-k2b | 90.0% | 79.9–95.3overlaps the top interval | Passed 54/60 | llama.cppvendor-k2b | 2026-09-10 |
| qwen38-27b-q3kxl-chad | 85.0% | 73.9–91.9 | Passed 51/60 | llama.cppextraction | 2026-09-18 |
| reap-60b-mlx-4bit-uniform | 85.0% | 73.9–91.9 | Passed 51/60 | mlxextraction | 2026-09-11 |
| bonsai2-27b-pq2 | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-18 |
| bonsai2-27b-ptq1 | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-18 |
| BTL-4 (Q4_K_M)btl-4-q4km | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-15 |
| Gemma 4 12Bgemma4-12b | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-09 |
| gemma4-12b-b10893 | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-12 |
| gemma4-12b-midrange | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-10 |
| qwen3-8b-midrange | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-10 |
| qwen3.8-27b-q4_0 | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-11 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-14 |
| qwen38-27b-np1 | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-10 |
| qwen38-gsqrco-iq3s | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-10 |
| reap-60b-20260910 | 83.3% | 72.0–90.7 | Passed 50/60 | llama.cppextraction | 2026-09-10 |
| edge0-35b | 81.7% | 70.1–89.4 | Passed 49/60 | llama.cppextraction | 2026-09-16 |
| reap-60b | 81.7% | 70.1–89.4 | Passed 49/60 | llama.cppextraction | 2026-09-09 |
| reap-60b-gguf-q4km | 81.7% | 70.1–89.4 | Passed 49/60 | llama.cppextraction | 2026-09-10 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 80.0% | 68.2–88.2 | Passed 48/60 | llama.cppextraction | 2026-09-15 |
| diffusiongemma-26b-mlx | 80.0% | 68.2–88.2 | Passed 48/60 | mlxextraction | 2026-09-11 |
| ornith-1.5-9b | 80.0% | 68.2–88.2 | Passed 48/60 | llama.cppextraction | 2026-09-12 |
| minicpm5-2b-q8-np1 | 78.3% | 66.4–86.9 | Passed 47/60 | llama.cppextraction | 2026-09-17 |
| k2-36b-q4_0 | 76.7% | 64.6–85.6 | Passed 46/60 | llama.cppextraction | 2026-09-11 |
| k2-36b-extraction | 75.0% | 62.8–84.2 | Passed 45/60 | llama.cppextraction | 2026-09-09 |
| k2-7b | 75.0% | 62.8–84.2 | Passed 45/60 | llama.cppextraction | 2026-09-09 |
| minicpm5-2b-exl3 | 75.0% | 62.8–84.2 | Passed 45/60 | llama.cppextraction | 2026-09-17 |
| k2-36b-a4b-q4 | 73.3% | 61.0–82.9 | Passed 44/60 | llama.cppextraction | 2026-09-09 |
| k2-36b-ctx32k | 70.0% | 57.5–80.1 | Passed 42/60 | llama.cppextraction | 2026-09-09 |
| bonsai-27b-mlx2-cortext | 66.7% | 54.1–77.3 | Passed 40/60 | mlxextraction | 2026-09-14 |
| bonsai-27b-mlx2-m4pro | 66.7% | 54.1–77.3 | Passed 40/60 | mlxextraction | 2026-09-14 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 66.7% | 54.1–77.3 | Passed 40/60 | llama.cppextraction | 2026-09-14 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 65.0% | 52.4–75.8 | Passed 39/60 | llama.cppextraction | 2026-09-14 |
| glm-4.7-flash-q40 | 60.0% | 47.4–71.4 | Passed 36/60 | llama.cppextraction | 2026-09-12 |
| minicpm5-2b | 60.0% | 47.4–71.4 | Passed 36/60 | llama.cppextraction | 2026-09-15 |
| ternary-bonsai-8b | 60.0% | 47.4–71.4 | Passed 36/60 | llama.cppextraction | 2026-09-17 |
| k2-3.7b | 58.3% | 45.7–69.9 | Passed 35/60 | llama.cppextraction | 2026-09-09 |
| k2-3.7b-q8-precision | 56.7% | 44.1–68.4 | Passed 34/60 | llama.cppextraction | 2026-09-10 |
| k2-4b-bf16-precision | 56.7% | 44.1–68.4 | Passed 34/60 | llama.cppextraction | 2026-09-10 |
| lfm25-8b-a1b-midrange | 53.3% | 40.9–65.4 | Passed 32/60 | llama.cppextraction | 2026-09-10 |
| k2-3.7b-q4-precision | 50.0% | 37.7–62.3 | Passed 30/60 | llama.cppextraction | 2026-09-10 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 35.0% | 24.2–47.6 | Passed 21/60 | llama.cppextraction | 2026-09-14 |
| spark-x25-4b | 28.3% | 18.5–40.8 | Passed 17/60 | llama.cppextraction | 2026-09-10 |
Run log49 logged runs of this suite
Every logged run, newest first (49)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| qwen38-27b-q3kxl-chad | not recordedextraction | 85.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c a1f473e0d63ec1b3 | |
| bonsai2-27b-ptq1 | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 522c48da0b9474a3 | |
| bonsai2-27b-pq2 | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 25c89df1304600f0 | |
| minicpm5-2b-q8-np1 | not recordedextraction | 78.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c e59bb419db23bea7 | |
| minicpm5-2b-exl3 | not recordedextraction | 75.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c d51ee21c635d7430 | |
| ternary-bonsai-8b | not recordedextraction | 60.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 00939f8fd32a28ff | |
| minicpm5-2b-exl3 | not recordedextraction | 100.0% | Coverage 13/13 | g 4156dd3c13361e8bd 267d1954e225c5b6c c0f3ff2bda3a17e8 | |
| edge0-35b | not recordedextraction | 81.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 0955df95170a937b | |
| minicpm5-2b | not recordedextraction | 60.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 602e85a1c355f821 | |
| btl-4-q4km | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 34ecdb4bf4ded955 | |
| btl-4 | not recordedextraction | 80.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 11c6d6217d23f0b7 | |
| bonsai-27b-1bit | not recordedextraction | 35.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c b6b3f7a5739fbaf7 | |
| bonsai-8b-1bit | not recordedextraction | 65.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 49c73224f7b988ec | |
| bonsai-27b-mlx2-cortext | mlxextraction | 66.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c a3e244ab46faf1d3 | |
| bonsai-27b-mlx2-m4pro | mlxextraction | 66.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 14f4738d0f91fc84 | |
| ternary-bonsai-27b | not recordedextraction | 66.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 6a368a22cfb7fb02 | |
| qwen3.8-9b-distill | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 8481e2a6e1f3e60c | |
| gemma4-12b-b10893 | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 41030e0a832a129e | |
| ornith-1.5-9b | not recordedextraction | 80.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 3fba7b7b697d2bc6 | |
| glm-4.7-flash-q40 | not recordedextraction | 60.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c b33ed27337b5ef7e | |
| qwen3.8-27b-q4_0 | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 7bca9c0f60cfafa3 | |
| k2-36b-q4_0 | not recordedextraction | 76.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 4bc7f1a68df72179 | |
| reap-60b-mlx-4bit-uniform | mlxextraction | 85.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 5f382a39bf24731a | |
| diffusiongemma-26b-mlx | mlxextraction | 80.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 6220b53efc9f35b4 | |
| reap-60b-gguf-q4km | not recordedextraction | 81.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 5d5c05e03d9c61e2 | |
| reap-60b-20260910 | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 078d5fad4e6d0a0f | |
| spark-x25-4b | not recordedextraction | 28.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c ffe0798e394ad2c2 | |
| k2-4b-bf16-precision | not recordedextraction | 56.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 0faa5511a9576d51 | |
| k2-3.7b-q8-precision | not recordedextraction | 56.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 3529bab4b75f7502 | |
| k2-3.7b-q4-precision | not recordedextraction | 50.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 49e024d8aab08569 | |
| gemma4-12b-midrange | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 480148dc413153dc | |
| lfm25-8b-a1b-midrange | not recordedextraction | 53.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 089fdc592c6d831d | |
| qwen3-8b-midrange | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 07092184bce46ed0 | |
| qwen38-gsqrco-iq3s | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 92ea3b5723445268 | |
| bonsai-27b-ternary | not recordedextraction | 66.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c ef7f6527771769a7 | |
| qwen38-27b-np1 | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 2867e80b35de0e49 | |
| k2-36b-vendor-k2b | not recordedvendor-k2b | 90.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 19ff3eed9d7b1a47 | |
| k2-36b-vendor-k2 | not recordedvendor-k2 | 93.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c b7e765413a3dd2f2 | |
| k2-36b-extraction | not recordedextraction | 75.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c b92607640a462939 | |
| k2-36b-ctx32k | not recordedextraction | 70.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 7873f3430a2f042e | |
| k2-36b-a4b-q4 | not recordedextraction | 73.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 7873f3430a2f042e | |
| gemma4-12b | not recordedextraction | 83.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c dea865f1d077f0d8 | |
| k2-7b | not recordedextraction | 75.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c d3f015ffb0c32d4d | |
| k2-3.7b | not recordedextraction | 58.3% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c be9a405fa957d750 | |
| reap-60b | not recordedextraction | 81.7% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 6e88592c339b65ea | |
| gemma4-12b | not recordedextraction | 80.0% | Coverage 60/60 | g 27df090db4445c2cd 760936a5348b2ffdc 9749137aeec83ad6 | |
| k2-7b | not recordedextraction | 75.0% | Coverage 60/60 | g 27df090db4445c2cd 760936a5348b2ffdc 58c73d3c88ebea78 | |
| k2-3.7b | not recordedextraction | 86.7% | Coverage 60/60 | g 27df090db4445c2cd 760936a5348b2ffdc cc272a124c5d1351 | |
| qwen3-next-80b-reap-mlx-mixed36 | mlxextraction | 100.0% | Coverage 60/60 | g 27df090db4445c2cd 760936a5348b2ffdc 93c5f1d3b8c8e17c |