Methodology.

Three layers, kept separate

Every result is recorded in three parts. The measurement is what the model actually emitted. The evaluation is what a grader made of it. The policy is what we decided as a result. Keeping them apart means a change of mind about policy does not quietly rewrite history, and a grader bug does not masquerade as a model result.

Coverage is always reported next to accuracy. A suite that scored 90 percent on half its cases is not the same as one that scored 90 percent on all of them, and collapsing the two hides harness failures inside model results.

Locked datasets

Each suite freezes its cases under a version string. The freeze records a hash per case and a dataset identifier for the set. If the cases drift, the runner refuses to start rather than comparing today's model against yesterday's questions. Results carry the dataset identifier, the grader identifier and a configuration identifier, so any number can be traced back to the exact questions, the exact grading code and the exact server settings that produced it.

Graders prove themselves first

A grader has to pass its own fixtures before it is allowed to score a model. Each suite ships gold responses that must pass and named failures that must fail. They run before a single model is called, and the run aborts if any of them disagree. The fixtures are written as prose a model might actually emit, because a fixture copied out of the source document passes any grader including a broken one.

This exists because a batch of earlier bugs were all the same bug. The grader was wrong, not the model. The rule that came out of it was reactive: if output looks good but scores zero, check the grader first. Making it structural removed the dependence on somebody noticing.

Confidence, and when a suite stops working

Every score is reported with a 95 percent Wilson interval. This matters more than it sounds. On a 72 case suite, 97 percent and 89 percent have overlapping intervals, which means the models are not distinguishable however tidy the headline looks.

Fixed difficulty suites stop being useful once the field clears them. Two suites are built as staircases instead. Toolscale increases the number of available tools until selection breaks down, and chainwalk increases chain depth until the model loses its place. They report the rung reached rather than a pass rate, so they do not saturate.

Runtime validity

Two suites, grounding and nullschema, depend on constrained decoding. The server must apply the supplied schema. mlx_lm accepts the field and ignores it, which turns the constrained arm into the free arm and produces a clean looking number for a constraint that was never applied.

The runner now probes each server with a schema whose only legal output is a fixed sentinel. If the reply does not match, those suites refuse to run. The probe distinguishes a server that ignores schemas from one that is simply unreachable, and gives reasoning models enough tokens to finish thinking before it reads their output.

Enforcement turns out to be per model rather than per runtime. llama.cpp applies the grammar for some models and silently drops it for others, so the probe runs against the model under test, not against the server in general.

What invalidates a run

• The dataset drifted from its frozen version.
• A grader fixture failed.
• Coverage was incomplete, which usually means the server was still loading.
• The suite required constrained decoding and the server did not apply it.
• The model was not the one the result claims. Checking which weights are resident before interpreting a number is now a standing step, after a throughput result was attributed to the wrong model entirely.

Hardware

All measurements are taken on one machine, an Apple M1 Max with 64 GB of unified memory and roughly 400 GB per second of memory bandwidth. That bandwidth sets a hard ceiling on decode speed for dense models, because a dense model streams all of its weights for every token produced. Fifteen gigabytes of weights cannot exceed about 27 tokens per second on this machine, whatever the software does. Mixture of experts models are not bound the same way, because only the active experts are read per token.

Published benchmark figures from other machines are not comparable without checking the bandwidth. A figure of 43 tokens per second for a dense 27B model implies roughly 650 GB per second, which is a different class of hardware.