What makes a model reach for a tool

Method

Five arms, identical tool machinery throughout, varying only the framing. Availability states a tool exists. Conditional puts an if-then rule in the system prompt. Descriptive puts the same rule in the tool description. Need makes the tool hold a value the model cannot otherwise produce. Unmentioned offers the tool with no reference to it anywhere.

Grading

Scored on invocation, not on the answer. A model that calls the tool and then misuses the result fails contract, not this suite.

Caveat

Results move more with the serving runtime than with the model. The same weights scored 78.3 on one server and 35.0 on another, so any comparison must hold the runtime fixed.

Version

toolcall/1

Cases

60

Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.

Top of the suite Bonsai-27B (Q1_0) 100.0% 95% interval 94.0–100.0

Open in the run log →

1. 2026-09-18A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output
2. 2026-09-18PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free
3. 2026-09-17A controlled 2x2 confirms it: the 27B loses 7.3 points to 1-bit, the 8B loses nothing
4. 2026-09-17A destroyed model and an incapable one produce the same zero, so the harness now refuses to run without a coherence check
5. 2026-09-17PQ2_0 and PTQ1_0 cannot post-quantise a trained model, and selectivity does not rescue them
6. 2026-09-15IQ2_XXS took 27 points off world knowledge and 17 off tool invocation on identical weights
7. 2026-09-15MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites
8. 2026-09-13A 23% smaller IQ3_S build decodes 27% slower than Q4_0 on the same weights - the codebook costs more than the bytes it saves
9. 2026-09-11Badly-chosen mixed precision costs abstention; sensitivity-aware mixing protects itrevised
10. 2026-09-10Q8_0 and BF16 are behaviourally identical; full precision buys nothing
11. 2026-09-10Tool perception is binary and tracks the checkpoint, not size or vendor claims
12. 2026-09-10gemma4-12b does not act on rules placed in tool descriptions
13. 2026-09-09No model qualifies single-source evidence, even when explicitly told to
14. 2026-09-09Judge and tool-use scores are runtime-sensitive because the decisions sit on ~0.4 logprob margins
15. 2026-09-09oMLX degrades models it has no native kernel for, by up to 58 points
16. 2026-09-09The serving runtime changes tool-calling results more than the model does
17. 2026-09-09Models call tools when given an if-then rule, not when told a tool exists