MLX versus llama.cpp.

The short version

Five models were run under both runtimes on the same machine. On the one suite where the comparison is valid, the two runtimes are indistinguishable. On the other two suites the numbers look like a clear MLX win, and that win is an artefact: MLX was given an easier task and nobody noticed for two weeks.

screening, matched or near-matched quantisation
                        MLX          GGUF
granite-4.2-30b    91.7  /////////   98.3  //////////
qwen3-next-80b     96.7  //////////  91.7  /////////
glm-4.7-flash      96.7  //////////  95.0  //////////
ornith-1.5-35b     88.3  /////////   95.0  //////////   (bit depths differ)

Findings

1. The choice of MLX runtime matters more than the choice between MLX and llama.cpp. rapid-mlx beats llama.cpp on every workload shape; mlx_lm loses on every one.
2. On screening, the only suite both runtimes can be compared on, differences are within overlapping confidence intervals in every pair.
3. On grounding and nullschema, MLX appears to win by wide margins. It does not. Those suites send a schema that mlx_lm accepts and ignores.
4. The tell was a perfect score. Granite scored 100 percent on nullschema under MLX against 90.3 under llama.cpp, with all three experimental arms identical.
5. Schema enforcement turns out to vary by model, not just by runtime.
6. Throughput differences are real and small on dense models, and the published figures people compare against are usually from different hardware.

What was held constant

Where the comparison is valid

screening needs no constrained decoding, so both runtimes get the same task.

Sixty cases each, so a 95 percent interval spans roughly nine points. Every pair overlaps. Two of the four favour GGUF and two favour MLX, which is what noise looks like.

The ornith row is included and labelled rather than dropped. Its GGUF build is 6-bit against a 4-bit MLX build, so any difference there includes bit depth and cannot be attributed to the runtime.

Where it is not valid, and why

grounding and nullschema both send an OpenAI-style response_format with a JSON schema. llama.cpp applies it as a sampling grammar. mlx_lm accepts the field and generates freely: response_format and json_schema appear zero times in its 1,904-line server.py.

nullschema exists to test whether forbidding absence causes a model to invent values. It runs three arms over the same documents: a required non-nullable field, a type union including null, and an omittable key. Under MLX all three arms collapse into one, because none of the schemas are applied.

The signature is visible in the raw counts.

granite-4.2-30b, nullschema, by experimental arm

MLX     forbid 24/24    permit 24/24    optional 24/24     identical
GGUF    forbid 22/24    permit 21/24    optional 22/24     arms differ

Fifteen result files across eight model runs were affected. They are still on this site, labelled unenforced rather than deleted, because a measurement of the wrong thing is not the same as no measurement.

Enforcement varies by model, not by runtime

Having found that MLX never enforces, the obvious assumption was that llama.cpp always does. It does not. Probing each model with a schema whose only legal output is a fixed sentinel string:

granite-4.2-30b     returns {"v": "__sentinel__"}     enforced
qwen3-coder-30b     returns {"v": "__sentinel__"}     enforced
muse-glimmer-30b    returns  blue                     NOT enforced

So the probe has to run against the model under test, not against the server. Muse Glimmer's nullschema score is unobtainable on either runtime, and is recorded as a gap rather than a number.

The runtime matters more than the framework

Measured on one model, granite-4.2-30b at 4-bit on both sides, identical prompts, identical token counts, four workload shapes. Only the runtime varies.

rapid-mlx wins every scenario. mlx_lm loses every scenario. The spread between the fastest and slowest MLX runtime is larger than the spread between MLX and llama.cpp.

That matters because every earlier comparison here used mlx_lm, which is the reference implementation rather than the fast one. Asking whether MLX beats llama.cpp is the wrong question. mlx_lm does not; rapid-mlx does, by 35 percent on decode-heavy work.

The advantage narrows to 9 percent on prefill stress, so it is a decode gain rather than a prompt-processing one.

Throughput has no consistent winner either

An earlier version of this work concluded that MLX is faster on dense architectures and llama.cpp faster on hybrids. That generalisation came from one model, with mismatched quantisation, and does not survive contact with the rest of the data.

An independent benchmark on the same machine specification, a Mac Studio M1 Max with 64 GB and 24 GPU cores, found llama.cpp ahead on both models it tested: 22.2 against 17.2 tok/s on a dense 12B, and 41.4 against 35.6 on a 30B mixture of experts (famstack.dev).

Three dense models now point in different directions. The only bit-matched dense pair here is a tie. The reasonable conclusion is that runtime advantage is a property of the specific model and its quantisation, not of the architecture class, and that any claim otherwise needs a bit-matched pair behind it.

What did not help

Mixed-precision quantisation. Two builds were made from the same bf16 source using mlx_lm convert --quant-predicate. They are smaller and scored lower on the only comparable suite: 93.3 percent for both mixed builds against 96.7 for plain 4-bit and 8-bit. Perplexity is still pending and may yet redeem them.

Speculative decoding with a published draft. A 1.5 GB DFlash draft paired with a dense 30B target ran at 2.8 tok/s against a 9.6 tok/s baseline, a 3.4 times slowdown, with 10.8 percent draft acceptance. Speculative decoding needs roughly 60 to 80 percent acceptance to pay for itself. That test is being repeated with a matched pair from one publisher, because the first attempt used a target and a draft from different repositories.

mlx_lm.dynamic_quant. Fails on mixture-of-experts models entirely. Sensitivity estimation backpropagates through a gather over experts, which MLX cannot differentiate.

The near miss

The KV cache cost of three models was predicted from their configuration files using the standard formula, two times layers times kv-heads times head-dim. Granite matched almost exactly, 0.250 predicted against 0.236 measured. The other two were wrong by six and seven times, and one of those errors was published here before the ladder was run and had to be corrected.

The lesson is narrow and worth stating plainly. A configuration file describes the architecture, not the runtime behaviour. Models using windowed or compressed attention report head counts that do not predict their cache. Run the ladder.