Working drawing 02 · Suites← Benchmarks
15 suites.
Local model benchmarking on a single 64 GB Mac. Every suite locked, every dataset hashed, graders that must prove themselves before they score anything.
- Suites
- 15
- Recorded runs
- 582
- Models measured
- 159
Measurements taken on an Apple M1 Max with 64 GB of unified memory.
Every number here comes from a locked dataset with a recorded grader. Where a measurement is not valid, it is marked rather than removed.
Each suite below answers one question. Click through for the method, the grading rule, the known caveats, and every run recorded against it.
- synthesis/2Turning tool results into an answer
Can a model synthesise a set of tool results into a usable, attributed answer?
- screening/1Second-pass screening
Can a second model catch the errors a grounding filter lets through?
- grounding/1Grounding and output mode
Does the mitigation work, and what does each output mode cost?
- nullschema/1Schema shape and abstention
Is it the schema that causes fabrication, or is it forbidding absence?
- tagger/2Tagger contract compliance
Does the model produce tags that satisfy the contract?
- tagger-nullable/1Tagger with absence permitted
Does allowing empty fields change tagging behaviour?
- epistemics/4Given versus prior knowledge
Does the model answer from the document it was given, or from what it already knows?
- enrichment/3Memory layer enrichment
Can a local model do the enrichment work the memory layer needs?
- tools/1Tool and skill selection
Given a request and a set of tools, does the model pick the right one and chain correctly?
- toolscale/1Tool selection as a staircase
How many tools can the model choose between before it picks the wrong one?
- chainwalk/3Long horizon chaining
How deep can the model chain tool calls before it loses its place?
- contract/2Following a convention it has to go and get
Will a model follow an output convention it can only obtain by asking for it?
- toolcall/1What makes a model reach for a tool
Is a model unable to call tools, or does it not perceive a need to?
- shapegen/2Build to an unseen contract
Can a model write a component to a composition contract it has never seen?
- sitebuild/1Agentic coding harness
Can a model build a working site in a real project with tools and a build it can run?
- needleneedle
Runs recorded, no published method.
- worldviewworldview
Runs recorded, no published method.
Seven models measured side by side in the August–September programme, every figure read straight out of the result files. Bars are pure CSS, so they print and they read at 320px. chainwalk omitted: n=2 for every model, never a usable sample.
Throughput, measured on a quiet box
300-token generations, three runs each, nothing else resident. Contention on this machine has produced a 2.6× error, so a number taken under load is not a number.
Cost of a seat
Weights plus the KV cache llama.cpp reserves up front. KV per token spans 96× across this fleet, so file size ranks seats wrongly.
Where each one breaks
More ink is better. A dot means the suite has not been run against that model. A model can lead one column and be worst in another — that is the whole argument for a routed fleet rather than one model.
| Model | nullschema | worldview | grounding | epistemics | toolcall | contract | synthesis | enrichment | screening | tagger | tag-null |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Bonsai 8B 1-bit | 19.4 | 40 | 55.2 | 80.8 | 0 | 20.8 | 65 | 83 | 86.7 | 80.3 | 86.3 |
| Ornith 35B-A3B | 36.1 | 85 | 76 | · | 80 | 66.7 | · | · | · | · | · |
| BTL-4 Q4_K_M | 44.4 | 55 | 76.8 | 100 | 93.3 | 70.8 | 83.3 | 98.8 | 91.7 | 96.4 | 94.1 |
| Qwen3.8-9B distill | 18.1 | 40 | 67.2 | 95.8 | 0 | 25 | 83.3 | 95.9 | 91.7 | 94.4 | 96.3 |
| BTL-4 IQ2_XXS | 36.1 | 62.5 | 76.8 | 100 | 76.7 | 64.6 | 80 | 98.7 | 95 | 96.5 | 96.2 |
| Bonsai 27B 1-bit | 18.1 | 40 | 67.2 | 87.5 | 100 | 25 | 35 | 95.1 | 96.7 | 91.7 | 95 |
| Bonsai 27B ternary | 27.8 | 42.5 | 68.8 | 85.4 | 100 | 75 | 66.7 | 88.4 | 93.3 | 96.2 | 97.9 |