What does each context rung actually cost in resident memory, and where is the safe ceiling?
KV cache scales with layers x kv-heads x ctx, NOT with active parameters. Decode speed is identical at every rung, context costs memory, never throughput. UPDATED 23 Aug: gemma4-31b being worst-in-fleet (+9.6GB to 128k) is about ITS 16 KV heads across 10 full-attention layers, not the Gemma 4 family. gemma4-12b holds 262144, twice the context, in 12.07GB, and at 131072 costs 10.07GB against qwen3-8b's 22.7GB. A 12B dense at less than half the price of an 8B dense at the same window. Note the caveat: per-token predictions from config arithmetic missed in BOTH directions across this family (E2B on target, E4B 2.3x over, 12B 0.5x under), so the ladder is the fact and the arithmetic is not.
- Method
- Context-window ladder, find the safe ceiling by measuring, not by arithmetic.
- Source
- ctx_ladder.py Context-window ladder, find the safe ceiling by measuring, not by arithmetic.
MLX versus llama.cpp
holds upWhich runtime is faster once you normalise for bits per weight?
MLX is ~1.45x faster. MLX with no speculation beats llama.cpp with MTP. The runtime is worth more than any speculation scheme.
- Method
- Qwen3.8-27B: llama.cpp GGUF vs MLX, same model, same prompts.
- Source
- runtime_ab.py Qwen3.8-27B: llama.cpp GGUF vs MLX, same model, same prompts.
Quantisation quality
holds upDoes a 4-bit quant still do the job?
GGUF k-quant scored 28/30 on extraction, categorisation and tagging. The MLX 4-bit run returned 0/30, all UNMEASURED, not all wrong. Validity gating is why that distinction survived.
- Method
- Quality A/B: muse-glimmer GGUF (kquant-dynamic) vs MLX 4-bit.
- Source
- quant_quality.py Quality A/B: muse-glimmer GGUF (kquant-dynamic) vs MLX 4-bit.
Needle in a haystack, retrieval at full window
holds upAllocating a long context is not the same as using one. Does the model actually RETRIEVE across its full native window?
gemma4-e2b and gemma4-e4b are 9/9 each, every length, every depth, including 126,976 tokens. The sliding-window architecture was the obvious thing to doubt (28 of E2B's 35 layers see only 512 tokens, 20 don't compute their own KV) and it retrieves flawlessly anyway. The cost is entirely PREFILL: ~1000 t/s at 8k falls to ~170 t/s at 127k, roughly 3x worse per token. Decode stays flat, so 'context costs memory, not throughput' holds for decode and NOT for prefill, which is the axis that governs agentic work at 31:1. The three older models here show NO MEASURED CASES, not zero scores: 36 rows with empty completions from the max_tokens bug, excluded rather than counted as misses.
- Method
- Aggregate results_longctx/*.jsonl into results_longctx.json for report.py.
- Source
- summarise_longctx.py Aggregate results_longctx/*.jsonl into results_longctx.json for report.py.
Fleet baseline, controlled llama-bench
holds upWhat are prefill and decode actually worth on this box, measured without prompt-cache state, chat templates or thinking variance?
Quantisation does NOT reliably buy decode speed. Qwen3.5-9B decodes at ~31 t/s across IQ4_NL, Q4_K_M and Q8_0 -- a 1.9x range in bytes per token for a 4% range in throughput. Ornith Q6_K really is faster than Q5_K_M (50.3 vs 45.2, 11.3%) while being 15% larger, so that anomaly survives a clean measurement. Prefill splits the fleet 6x by active parameters: ~3B-active MoEs run 519-678 t/s, dense 27-31B run 93-101 t/s.
- Method
- Fleet-wide llama-bench baseline, controlled prefill and decode for every GGUF.
- Source
- bench_fleet.py Fleet-wide llama-bench baseline, controlled prefill and decode for every GGUF.