Suite · toolscale← Suites
Tool selection as a staircase
How many tools can the model choose between before it picks the wrong one?
- Cases
- 40
- Scored runs
- 45
- Models
- 45
- Top score
- 100.0%
SpecificationMethod · grading · caveat
- Method
- The number of available tools increases step by step. The task stays the same. The suite records the rung at which selection breaks down.
- Grading
- Scored by the highest rung cleared, not by a fixed pass rate.
- Caveat
- Built to avoid saturation. A model that clears every rung tells you the staircase needs more steps, not that the model is finished.
- Version
toolscale/1- Cases
- 40
Recorded runs45 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite Agents-A1 35B 100.0% 95% interval 91.2–100.0
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| Agents-A1 35Bagents-a1-35b | 100.0% | 91.2–100.0 | Passed 40/40 | llama.cppagentic-fast | 2026-08-24 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-14 |
| bonsai-27b-mlx2-cortext | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | mlxagentic-fast | 2026-09-14 |
| bonsai-27b-mlx2-m4pro | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | mlxagentic-fast | 2026-09-14 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-14 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-15 |
| BTL-4 (Q4_K_M)btl-4-q4km | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-15 |
| diffusiongemma-26b-mlx | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | mlxextraction | 2026-09-11 |
| Gemma 4 E2Bgemma4-e2b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-08-24 |
| gemma4-e4b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-08-24 |
| glm-4.7-flash-q40 | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-12 |
| k2-3.7b-q4-precision | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-10 |
| k2-3.7b-q8-precision | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-10 |
| k2-4b-bf16-precision | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-10 |
| K2-Horizon-3.7B (IFM)k2-horizon-3.7b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-09 |
| k2-horizon-7b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-09 |
| minicpm5-2b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-15 |
| neohorse-1-4b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-13 |
| ornith-35b-a3b-q40 | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-12 |
| qwen3-8b-midrange | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-10 |
| qwen3.5-4b-base | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-13 |
| qwen3.8-27b-q4_0 | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-11 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-14 |
| qwen38-27b-np1 | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-10 |
| qwen38-gsqrco-iq3s | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppextraction | 2026-09-10 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-14 |
| ternary-bonsai-8b | 100.0% | 91.2–100.0overlaps the top interval | Passed 40/40 | llama.cppagentic-fast | 2026-09-17 |
| bonsai2-27b-pq2 | 97.5% | 87.1–99.6overlaps the top interval | Passed 39/40 | llama.cppagentic-fast | 2026-09-18 |
| bonsai2-27b-ptq1 | 97.5% | 87.1–99.6overlaps the top interval | Passed 39/40 | llama.cppagentic-fast | 2026-09-18 |
| gemma4-12b-midrange | 95.0% | 83.5–98.6overlaps the top interval | Passed 38/40 | llama.cppextraction | 2026-09-10 |
| ornith-1.5-9b | 95.0% | 83.5–98.6overlaps the top interval | Passed 38/40 | llama.cppextraction | 2026-09-12 |
| qwen3-next-80b-ag | 95.0% | 83.5–98.6overlaps the top interval | Passed 38/40 | llama.cppextraction | 2026-09-07 |
| reap-60b-20260910 | 95.0% | 83.5–98.6overlaps the top interval | Passed 38/40 | llama.cppextraction | 2026-09-10 |
| reap-60b-mlx-4bit-uniform | 95.0% | 83.5–98.6overlaps the top interval | Passed 38/40 | mlxextraction | 2026-09-11 |
| spark-x25-4b | 95.0% | 83.5–98.6overlaps the top interval | Passed 38/40 | llama.cppextraction | 2026-09-10 |
| Gemma 4 12Bgemma4-12b | 90.0% | 76.9–96.0overlaps the top interval | Passed 36/40 | llama.cppagentic-fast | 2026-08-24 |
| k2-36b-a4b-q4 | 90.0% | 76.9–96.0overlaps the top interval | Passed 36/40 | llama.cppextraction | 2026-09-09 |
| k2-36b-ctx32k | 90.0% | 76.9–96.0overlaps the top interval | Passed 36/40 | llama.cppextraction | 2026-09-09 |
| k2-36b-q4_0 | 90.0% | 76.9–96.0overlaps the top interval | Passed 36/40 | llama.cppextraction | 2026-09-11 |
| qwen3-next-80b-reap-ag | 90.0% | 76.9–96.0overlaps the top interval | Passed 36/40 | llama.cppagentic-fast | 2026-09-08 |
| k2-36b-np1 | 87.5% | 73.9–94.5overlaps the top interval | Passed 35/40 | llama.cppextraction | 2026-09-10 |
| reap-60b-gguf-q4km | 87.5% | 73.9–94.5overlaps the top interval | Passed 35/40 | llama.cppextraction | 2026-09-10 |
| lfm25-8b-a1b-midrange | 82.5% | 68.0–91.3overlaps the top interval | Passed 33/40 | llama.cppextraction | 2026-09-10 |
| qwen3.6-35b-ag | 80.0% | 65.2–89.5 | Passed 32/40 | llama.cppextraction | 2026-09-07 |
| qwen-agentworld-35b | 72.5% | 57.2–83.9 | Passed 29/40 | llama.cppextraction | 2026-09-07 |
Run log46 logged runs of this suite
Every logged run, newest first (46)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| bonsai2-27b-ptq1 | not recordedagentic-fast | 97.5% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 92433e8a2c1d7275 | |
| bonsai2-27b-pq2 | not recordedagentic-fast | 97.5% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c c76ee9534d06b617 | |
| ternary-bonsai-8b | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 4a4dea00c5224630 | |
| minicpm5-2b | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 2400bd439331eb46 | |
| btl-4-q4km | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 83afd32cbd70262a | |
| btl-4 | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 962089edc90681de | |
| bonsai-27b-1bit | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 8ee047a8f4681484 | |
| bonsai-8b-1bit | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 6fc2b443c7d76dde | |
| bonsai-27b-mlx2-cortext | mlxagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c bf7830ad8b1e5adb | |
| bonsai-27b-mlx2-m4pro | mlxagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 52e205dd25e5567b | |
| ternary-bonsai-27b | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 13185468aa0f107f | |
| qwen3.8-9b-distill | not recordedagentic-fast | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 13a2f2f87c9bcdc8 | |
| qwen3.5-4b-base | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 634f8ac324d31747 | |
| neohorse-1-4b | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c b90d64d3c6c193a5 | |
| glm-4.7-flash-q40 | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 5383698d8f3d7c82 | |
| ornith-35b-a3b-q40 | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 9ed2b4b41dc68126 | |
| ornith-1.5-9b | not recordedextraction | 95.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 8105ecbc4488d189 | |
| qwen3.8-27b-q4_0 | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 7dedd451e841393d | |
| k2-36b-q4_0 | not recordedextraction | 90.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 5467852ddcdb11c6 | |
| reap-60b-mlx-4bit-uniform | mlxextraction | 95.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c a58faee14a0f3ed6 | |
| diffusiongemma-26b-mlx | mlxextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 40a7e393e161d98f | |
| reap-60b-gguf-q4km | not recordedextraction | 87.5% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 437c75c01b4063cd | |
| reap-60b-20260910 | not recordedextraction | 95.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c ce60a8f4e40a948a | |
| spark-x25-4b | not recordedextraction | 95.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 266788017096e178 | |
| k2-4b-bf16-precision | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c e21f6ac8887f7f5d | |
| k2-3.7b-q8-precision | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 1eedcfdecea95e86 | |
| k2-3.7b-q4-precision | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 8bb7dfa4a8ce435f | |
| gemma4-12b-midrange | not recordedextraction | 95.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c bf8723aa88c0d4b7 | |
| lfm25-8b-a1b-midrange | not recordedextraction | 82.5% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 4727b9e997fb1a2e | |
| qwen3-8b-midrange | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c d27e6f32d963b586 | |
| qwen38-gsqrco-iq3s | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 3177b556489558c1 | |
| bonsai-27b-ternary | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 83edc9129d9d77cd | |
| k2-36b-np1 | not recordedextraction | 87.5% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 6e72c541e7fd804e | |
| qwen38-27b-np1 | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 2b5ec5e83a2c8961 | |
| k2-36b-ctx32k | not recordedextraction | 90.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c b2321bc042bae630 | |
| k2-36b-a4b-q4 | not recordedextraction | 90.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c b2321bc042bae630 | |
| k2-horizon-3.7b | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 4535019c467a8444 | |
| k2-horizon-7b | not recordedextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 62b1c628753b16b6 | |
| qwen3-next-80b-reap-ag | not recordedagentic-fast | 90.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 2b03adebd9317650 | |
| qwen3.6-35b-ag | llama.cppextraction | 80.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c f995b25bd0b360cf | |
| qwen3-next-80b-ag | llama.cppextraction | 95.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c 18366d2ac6e48e7c | |
| qwen-agentworld-35b | llama.cppextraction | 72.5% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c edcc1fd1f9bd35f0 | |
| agents-a1-35b | llama.cppagentic-fast | 100.0% | Coverage 40/40 | g 1bae1f3e1b606eaed b24abb108c409178c f55610aeacd8ae8c | |
| gemma4-12b | llama.cppagentic-fast | 90.0% | Coverage 40/40 | g 1bae1f3e1b606eaed b24abb108c409178c 42412bb32a456ced | |
| gemma4-e4b | llama.cppagentic-fast | 100.0% | Coverage 40/40 | g 1bae1f3e1b606eaed b24abb108c409178c 64d002411f012d3c | |
| gemma4-e2b | llama.cppagentic-fast | 100.0% | Coverage 40/40 | g 1bae1f3e1b606eaed b24abb108c409178c 3860ab39a73cff08 |
Findings7 that name this suite
- 2026-09-18A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output
- 2026-09-18PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free
- 2026-09-15MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites
- 2026-09-13A 23% smaller IQ3_S build decodes 27% slower than Q4_0 on the same weights - the codebook costs more than the bytes it saves
- 2026-09-10Q8_0 and BF16 are behaviourally identical; full precision buys nothing
- 2026-09-10Tool perception is binary and tracks the checkpoint, not size or vendor claims
- 2026-09-10toolscale's low levels are harder than its high levels