{
 "generated": "2026-09-20",
 "findings": [
  {
   "id": "runtime-does-not-affect-throughput",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "llama.cpp, mlx_lm and rapid-mlx decode at the same speed; only the format differs",
   "claim": "On identical weights all three runtimes are within 6%. The large differences attributed to runtime choice on Apple Silicon are quantisation-format differences in disguise.",
   "evidence": "Qwen3.5-9B, same weights: GGUF Q4_K_M 21.60 tok/s, GGUF Q4_0 44.11, mlx_lm affine-g64 45.30, rapid-mlx affine-g64 46.64. llama.cpp's DEFAULT format is the outlier at half speed; its good format matches both MLX runtimes exactly. rapid-mlx adds no measurable serving overhead over in-process mlx_lm."
  },
  {
   "id": "format-finding-does-not-transfer-to-mlx",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "The GGUF format result does not transfer to MLX -- symmetric quantisation is the WORST there",
   "claim": "Q4_0 (symmetric) beats Q4_1 (asymmetric) in GGUF. In MLX the ordering reverses: affine (asymmetric) beats mxfp4 (symmetric). The GGUF result is a llama.cpp K-quant implementation detail, not a property of quantisation schemes.",
   "evidence": "Qwen3.5-9B converted three ways and run on nullschema: MLX affine group-64 43/72, affine group-32 33/72, mxfp4 25/72. All three within 7% on speed (45.30 / 44.86 / 42.40 tok/s), so the differences are not throughput artefacts. In GGUF on k2-36b the same axis ran the other way: Q4_0 +9/-1 against Q4_K_M, Q4_1 +1/-4. Also: more scales did not help -- affine group-32 has twice the scales of group-64 and scored lower."
  },
  {
   "id": "runtime-changes-refusal-phrasing",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Runtime changes HOW a model phrases a refusal, and nullschema scores that as fabrication",
   "claim": "The same weights decline by writing 'not mentioned' under llama.cpp and by writing an empty string under MLX. The grader accepts only the latter, so a formatting difference reads as a 2.5x capability difference.",
   "evidence": "Qwen3.5-9B absent-case failures: GGUF Q4_0 24 of 24 were the phrase 'not mentioned'; GGUF Q4_K_M 17 of 19; MLX affine-g64 1 of 3; MLX affine-g32 0 of 3. Scores 6.9% / 23.6% / 59.7% / 45.8%. The grader classifies a present, non-empty, not-in-document value as `fabricated`, which is the wrong category for a correct refusal. k2-36b and qwen3.8-27b are unaffected -- zero of their failures are verbal declines -- so their comparisons stand. Analyser at benchmarks/tools/decline_forms.py reads stored responses without touching grader_id."
  },
  {
   "id": "spec-draft-p-min-default-is-harmful",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "llama.cpp's --spec-draft-p-min defaults to 0.00, which guarantees wasted drafts",
   "claim": "Speculative decoding drafts unconditionally at the default, paying a full verify pass for tokens that were never going to be accepted. Any speculation benchmark run at the default is measuring waste.",
   "evidence": "qwen3.8-27b MTP: acceptance 0.409 at the default, 1.000 with --spec-draft-p-min 0.7. k2-36b with a 3.7B draft: 0.562 at the default, 0.960 with a 0.7 floor. Throughput on k2 improved from 14.34 to 17.92 tok/s prose with the floor -- still half the 34.76 baseline, because acceptance was never the binding constraint there."
  },
  {
   "id": "dflash-draft-works-on-structured-output",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "A purpose-built draft gives 1.13x on structured output and loses on prose",
   "claim": "Speculative decoding pays on a dense target when the draft is both tiny and trained for the job. Every method tested helps JSON and hurts prose, because draft runs are short-lived in open text.",
   "evidence": "z-lab DFlash2, a 5-layer draft matching Qwen3.8-27B's vocab and hidden size, converted to GGUF (llama.cpp's converter has first-class DFlash support via --target-model-dir) and quantised to 1.0 GB -- 7% of the 14.6 GB target. Acceptance 0.972. JSON 15.68 vs 13.87 baseline (1.13x); prose 12.43 vs 14.48 (0.86x). A Q8_0 draft at 1.9 GB scored identically (0.986 acceptance, 15.71 JSON), so the cheaper draft wins on footprint. For comparison: MTP 1.066x on JSON, n-gram 1.09x, a 2.5 GB general-purpose draft 1.00x."
  },
  {
   "id": "benchmark-tok-s-is-an-empty-context-artefact",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Benchmark tok/s figures are empty-context artefacts; real decode falls 2.7x with context",
   "claim": "llama-bench reports decode at zero context. Under realistic agentic context the same model runs at a third of that, and the published figure is never what a user experiences.",
   "evidence": "k2-36b Q4_0, llama-bench tg64: 49.67 tok/s. Measured through the server on real requests: 43.8 tok/s decode at empty context, 28.8 at ~4k, 16.0 at ~15k. End-to-end at 15k was 3.26 tok/s on a cold cache because prefill dominates; with a warm prefix the same request returns in 5.5s, so prefix caching rescues multi-turn use but not first contact. Also: llama.cpp's `timings` fields are unreliable -- the same workload reported prefill at 226 tok/s and 21.7 tok/s in different runs. Use wall time."
  },
  {
   "id": "q4-0-beats-q4-k-m-on-metal",
   "date": "2026-09-14",
   "status": "revised",
   "title": "Q4_0 decodes up to 2x faster than Q4_K_M at the same bit-width, and emits valid JSON more reliably",
   "claim": "REVISED -- scoped to DENSE models; unvalidated on MoE. Q4_0 and Q4_K_M are both ~4.5 BPW. Q4_0 decodes 1.25-2.04x faster and produces parseable output under constrained decoding far more often. It does NOT improve abstention judgement -- that was a formatting artefact, corrected below.",
   "evidence": "Identical weights, one variable. granite-guardian-8b 36.84 -> 52.22 tok/s; Qwen3.5-9B 31.24 -> 43.84 at tg64 and 21.60 -> 44.11 at tg128; qwen3.8-27b 9.79 -> 15.18; k2-36b 39.85 -> 49.67. Q4_K_M DEGRADES with output length while Q4_0 holds flat, so the gain is larger in real use than short benchmarks show (1.40x at 64 tokens, 2.04x at 128). On k2-36b's nullschema the headline 54.2 -> 65.3 decomposes as: malformed 12 -> 3, misattributed 21 -> 22. The gain is entirely fewer unparseable responses under response_format; the grader marks those enforcement_absent, i.e. arguably void runs. Judgement was unchanged. Mechanism: Q4_K_M uses 256-element super-blocks with 6-bit quantised scales and promotes some tensors to Q6_K; Q4_0 is 32-element blocks with a full fp16 scale, two weights per byte. SCOPE (2026-09-14): every model in this evidence set whose architecture is recorded is DENSE -- granite-guardian-4.1-8b and qwen3.8-27b confirmed from their GGUF headers; k2-36b is no longer on disk and its architecture is not recorded anywhere. The ratio above is therefore a dense-model result and was stated without scope. Suggestive counter-observation, NOT a measurement: two 35B-A3B MoEs serving side by side gave agents-a1 (Q4_K_M) 49.1 t/s against ornith-1.5 (Q4_0) 53.4 t/s, a ratio of 1.10x. Those are DIFFERENT WEIGHTS, so it is not a quantisation comparison and cannot confirm anything. No MoE model exists on this disk in two formats, so the clean same-weights test has not been run.",
   "impact": " The mechanism predicts the gap: byte misalignment is paid per byte actually read, and an MoE reads only its active experts (8 of 256 here, ~3B of 35B params), so the absolute penalty should shrink roughly in proportion to sparsity. Predicted, not measured.",
   "action": " Do NOT apply this ratio when sizing or selecting an MoE seat -- it is unvalidated there. To close the gap, requantise one MoE to a second format (agents-a1 Q4_K_M -> Q4_0 is the cheapest path; decode speed depends on layout, not on requantisation quality loss) and run llama-bench on both. RESOLVED 2026-09-15: the MoE question is now measured, and the answer differs by format family -- see iq-codebook-penalty-transfers-to-moe-kquant-does-not. K-quant penalty falls to 1.10x on a sparse MoE; IQ codebook penalty RISES to ~2.1x. Use that finding, not this one, when sizing an MoE seat."
  },
  {
   "id": "byte-alignment-determines-decode-speed",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Byte alignment, not bit-width, determines quantised decode speed",
   "claim": "Formats storing an integer number of weights per byte decode far faster than formats whose weights span byte boundaries, independent of how many bits they use.",
   "evidence": "k2-36b, same weights, five formats: Q4_0 (4.5 BPW, 2 weights/byte) 49.67 tok/s; Q4_1 (5.0 BPW, 2 weights/byte) 48.82; Q5_0 (5.5 BPW, spans bytes) 35.20; Q5_1 (6.0 BPW, spans bytes) 34.48; Q4_K_M (4.5 BPW, super-blocks) 39.85. Q4_1 carries half a bit more than Q4_0 and loses under 2%; Q5_0 carries one more bit than Q4_1 and loses 28%. Clean two-group split with no overlap. Q5_0 and Q5_1 are strictly dominated -- larger AND slower than Q4_K_M."
  },
  {
   "id": "quant-gain-depends-on-source-format",
   "date": "2026-09-11",
   "status": "revised",
   "title": "Quantisation format changes output CONVENTIONS, not abstention judgement",
   "claim": "REVISED. The original claim -- that Q4_0 helps from uniform K-quants and hurts from dynamic quants -- read two different formatting artefacts as opposite capability changes. The measure of actual fabrication is unchanged by format in both directions.",
   "evidence": "Decomposing nullschema failures by type: k2-36b Q4_K_M -> Q4_0 moved malformed 12 -> 3 with misattributed 21 -> 22. qwen3.8-27b UD -> Q4_0 moved verbal-declines 6 -> 11 with misattributed 11 -> 11, identical. So k2 'improved' by emitting valid JSON more often and the 27B 'degraded' by phrasing refusals as text more often. Neither model changed how often it asserted a wrong value. A third model, Qwen3.5-9B, appeared to lose 16.7 points; 24 of 24 absent-case failures were the literal phrase 'not mentioned' -- a correct refusal in an unaccepted form."
  },
  {
   "id": "moe-decode-wastes-half-the-bandwidth",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "MoE decode reaches a third of dense memory bandwidth, and no runtime fixes it",
   "claim": "On Apple Silicon, MoE models achieve 22-36% of peak memory bandwidth during single-stream decode while dense models reach 59%. The gap is GPU occupancy, not bandwidth, and is identical across llama.cpp and MLX.",
   "evidence": "Bytes-per-token computed from each GGUF tensor table. gemma-4-12b dense 59% of 400 GB/s; gemma-4-26B-A4B (8 of 128) 36%; k2-36b-a4b (8 of 100) 31%; REAP qwen3next (10 of 384) 22%. The same REAP prune: llama.cpp 40.27 tok/s, MLX 40.07 tok/s -- 0.5% apart, both at 22%. Sparser routing is consistently worse, which is the direction the industry is moving."
  },
  {
   "id": "mlx-is-not-faster-than-llamacpp",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "The 'MLX is 3x faster on MoE' claim benchmarks Ollama, not llama.cpp",
   "claim": "Raw llama.cpp and raw mlx_lm decode at the same speed on the same weights. The widely repeated 3x figure uses Ollama as its baseline and measures wrapper overhead.",
   "evidence": "REAP qwen3next, same prune: llama.cpp Metal Q4_K_M 40.27 tok/s vs MLX 4-bit 40.07 tok/s. qwen3.8-27b: llama.cpp Q4_0 15.18 tok/s vs OptiQ/MLX 4-bit 13.90. The cited source reads '130 tok/s vs 43 tok/s for Ollama with the llama.cpp backend'."
  },
  {
   "id": "diffusion-is-compute-bound-on-apple-silicon",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Diffusion LLMs are compute-bound on Apple Silicon, so quantising them buys nothing",
   "claim": "Masked-diffusion decode cost tracks active parameters, not file size. Quantisation reduces memory traffic, which is not the constraint on this hardware, so it changes throughput by roughly zero.",
   "evidence": "LLaDA-MoE-7B-A1B at BF16 (13.7 GB) 341.30 ms/step vs Q4_K_M (4.2 GB) 343.29 ms/step -- 3.3x less memory traffic, 0.6% difference. The 4.6 GB dense LLaDA-8B runs 1501 ms/step against the 13.7 GB MoE's 341: the three times larger file is 4.4x faster because it activates 1B params instead of 8B. Diffusion trades compute for bandwidth; Apple Silicon offers ~52 FLOP/byte against a datacenter GPU's ~295."
  },
  {
   "id": "speculation-budget-is-active-params",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "On a sparse MoE, speculative decoding fails even at 96% draft acceptance",
   "claim": "Speculation requires draft_cost < target_cost x (k-1)/k. For an MoE the target cost is the ACTIVE read, not the file size, so sparsity removes the headroom speculation needs. Acceptance is not the binding constraint: a draft can predict almost perfectly and still lose.",
   "evidence": "k2-36b-a4b reads 3.35 GB/token; the smallest same-vocab draft, k2-3.7B, reads 2.90 GB -- 87% of the target. With --spec-draft-p-min 0.7, acceptance reached 0.960 (97/101) and throughput was still 17.59 tok/s prose / 18.75 json against a 34.76 / 32.47 baseline -- 0.51x. Depth 3 at p-min 0.8: 0.938 acceptance, 17.92 / 18.52. Predicted 0.70x from bytes alone; measured 0.51x, the difference being two-model overhead. On the DENSE qwen3.8-27b the same mechanism works but barely: MTP with p-min 0.7, measured against the same file with speculation off, gives 1.066x on JSON and 0.93x on prose. n-gram speculation (no draft model) gives 1.09x on JSON, nothing on prose. SEPARATE FINDING: --spec-draft-p-min defaults to 0.00, which drafts unconditionally and pays a full verify pass for tokens that were never going to be accepted. Setting a 0.7 floor took qwen3.8-27b MTP acceptance from 0.409 to 1.000 and k2 draft acceptance from 0.562 to 0.960. Any speculation benchmark run at the default is measuring wasted drafts."
  },
  {
   "id": "confidence-trace-is-not-a-fabrication-detector",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Per-token confidence flags hesitant errors and misses confident ones",
   "claim": "A per-position confidence trace isolates the decision token from the JSON scaffolding and catches uncertain answers, but fabrications committed at full confidence are invisible to it. It is a triage signal, not a correctness check.",
   "evidence": "Instrumented llama.cpp's masked-diffusion decoder to record the confidence each token was committed at. In structured extraction the scaffolding tokens sit at 1.000 and the abstention decision alone is uncertain (0.68). But asked for a VAT number from text containing only a company number, the model returned the company number at confidence 1.000. Separately, 91 of 481 tokens in a baseline generation were committed below 0.5 confidence and were precisely the tokens that made the output ungrammatical."
  },
  {
   "id": "prompt-framing-beats-bit-width-for-abstention",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Prompt framing outweighs a 2.7x bit-width range for abstention",
   "claim": "Whether a schema permits an empty answer moves abstention far more than quantisation does. On a diffusion model the permit arm was byte-identical from 2-bit to 8-bit while the forbid arm moved 23 points.",
   "evidence": "LLaDA-8B-Instruct, 13 frozen cases, five quantisations Q2_K to Q8_0. Permit arm (empty string allowed): 92.3% at every rung, with byte-identical extracted values on all 13 cases across the whole range. Forbid arm (non-empty required): 61.5 / 84.6 / 84.6 / 76.9 / 76.9. Detection never breaks -- the absent arm is 4/4 at every rung in both arms. What degrades is willingness to decline under instruction."
  },
  {
   "id": "mixed-precision-costs-abstention",
   "date": "2026-09-11",
   "status": "revised",
   "title": "Badly-chosen mixed precision costs abstention; sensitivity-aware mixing protects it",
   "claim": "REVISED 2026-09-11. The original claim -- that non-uniform precision degrades abstention at equal average bit-width -- fails on both halves. The bit-widths were never equal, and a principled mixed-precision scheme BEATS uniform quantisation on abstention. What matters is whether the allocation is sensitivity-aware, not whether it is uniform.",
   "evidence": "The original comparison used qwen3-next-80b-reap mlx-mixed36, which declares bits:4 but overrides 487 of 542 tensors DOWN to 3-bit -- a bit-width comparison wearing a mixed-precision label. Reconverting the same prune at mixed36's own base settings (4-bit, group 64, affine, no overrides) gave nullschema 84.7% vs 47.2%, but toolcall fell 78.3% -> 25.0%. Decisively, on qwen3.8-27b the direction reverses: its Unsloth Dynamic baseline (per-tensor overrides protecting sensitive tensors) scores nullschema 76.4%, and uniform Q4_0 on the same weights scores 69.4% -- 0 cases fixed, 5 broken, every one an abstention case (3 lure, 2 absent). Mixed precision chosen by sensitivity protects exactly the capability uniform quantisation loses."
  },
  {
   "id": "quantisation-tax-scales-with-capability",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "Quantisation damage is proportional to the capability present",
   "claim": "Losing precision harms a model that can abstain and does nothing to one that cannot. This inverts the usual 'larger models withstand quantisation better' reading on this capability.",
   "evidence": "nullschema, Q4_K_M to Q8_0, same weights each time: K2-3.7B +4.2, K2-7B -1.4, K2-36B-A4B +16.6 with 12 cases fixed and 0 broken, every case kind improving (absent 87.5 to 100, authority_lure 58.3 to 75.0, lure 16.7 to 37.5). The two small models sit at 36-43% and are flat across a 3x precision range; there is little capability there to damage."
  },
  {
   "id": "q8-is-the-ceiling-bf16-is-waste",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "Q8_0 and BF16 are behaviourally identical; full precision buys nothing",
   "claim": "Above Q8 there is no measurable difference on any suite. Memory is better spent on parameters.",
   "evidence": "K2-3.7B same weights: Q8_0 and BF16 differ on 0 of 72 nullschema cases, and produce identical scores on all five suites (toolcall 100/100, toolscale 100/100, worldview 50.0/50.0, nullschema 43.1/43.1, synthesis 56.7/56.7). K2-7B agrees (37.5/36.1/37.5 across Q4/Q8/BF16). Memory-matched: K2-7B BF16 at 16.8 GB scores nullschema 37.5% and worldview 30.0%; K2-36B Q4_K_M at 20.8 GB scores 54.2% and 52.5%."
  },
  {
   "id": "abstention-is-two-capabilities",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "Detecting emptiness and rejecting a plausible candidate are different capabilities",
   "claim": "Every model can say 'not present' when nothing is present. Almost none can reject a plausible wrong value that is present. Only the second discriminates.",
   "evidence": "nullschema by case kind. absent: spark-x2.5-4b 100% at 2.4 GB, gemma4-12b 100%, k2-36b 87.5%. lure: gemma4-12b 83.3%, k2-36b Q8 37.5%, spark 25.0%, k2-36b Q4 16.7%, bonsai-27b 12.5%. Observed in production the same day: K2-36B, given search results mentioning a real company at a similar domain, asserted that intershapes.com links to that company's privacy policy. The page has three external links and none is that."
  },
  {
   "id": "tool-perception-is-binary",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "Tool perception is binary and tracks the checkpoint, not size or vendor claims",
   "claim": "A model either reaches for an unannounced tool or it never does. Nothing between, and it correlates with neither parameter count nor agentic marketing.",
   "evidence": "toolcall/1, 60 cases: k2-3.7b (2.9 GB) 100%, bonsai-27b 100%, qwen3.8-27b 100%, k2-36b 100%; spark-x2.5-4b 0%, qwen3-8b 0%, lfm2.5-8b-a1b 1.7%. Spark-X2.5-4B is marketed for agentic tool use (tau3-bench 30.4, 'native compatibility with agent harnesses including Hermes') and scores 0/60 here while scoring 95% on toolscale. Its vendor claim concerns selection inside a harness loop; toolcall measures perception when nothing announces the tool."
  },
  {
   "id": "toolscale-levels-are-inverted",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "toolscale's low levels are harder than its high levels",
   "claim": "The staircase does not increase in difficulty. Low levels are almost entirely near-misses; high levels dilute them among unrelated filler. Weak models score worse at 4 tools than at 64.",
   "evidence": "Per level (4/8/16/32/64): lfm2.5-8b-a1b 5,6,7,7,8 of 8; k2-36b 6,7,6,8,8. Both are perfect at 64 and fail at 4. The suite always offers the target plus all its near-misses, then pads with unrelated tools. Consequence for MCP surface design: a large tool surface is an ingestion cost (~65 prompt tokens per schema, linear to 64 tools; ~80 s for 400 tools at 330 tok/s) and not a selection risk. Similar tool names are the risk."
  },
  {
   "id": "gemma-ignores-tool-descriptions",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "gemma4-12b does not act on rules placed in tool descriptions",
   "claim": "The same if-then rule scores 12/12 in the system prompt and 0/12 in the tool description on this model, which partially invalidates the docstring strategy for it.",
   "evidence": "toolcall/1 arms for gemma4-12b: availability 9/12, conditional 12/12, descriptive 0/12, need 12/12, unmentioned 0/12. Every other model tested scores 12/12 on both conditional and descriptive. Rules that must fire on gemma belong in the system prompt. See if-then-rule-drives-tool-invocation, which verified that Pydantic AI passes the full docstring into the tool description - the rule travels, but whether it lands is per-model."
  },
  {
   "id": "gemma-is-the-checker",
   "date": "2026-09-10",
   "status": "confirmed",
   "title": "The 6.7 GB tagger is the most epistemically reliable model in the fleet",
   "claim": "gemma4-12b leads both discriminating suites and is the only model that flags single-source evidence, at a fifth of the memory of the models it beats.",
   "evidence": "worldview 39/40 (97.5%) including single_source 7/8 - every other model scores 0/8 on that arm (see no-model-caveats-single-source-evidence, which was reached before gemma was measured on it). nullschema 63/72 (87.5%) with lure resistance 83.3%. Against k2-36b Q8 at 37.1 GB: 55.0% and 70.8%. Reproduced across two runs and two serving configurations."
  },
  {
   "id": "no-model-caveats-single-source-evidence",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "No model qualifies single-source evidence, even when explicitly told to",
   "claim": "Models act on the content of evidence, not its provenance tier. Asked to flag a claim supported by only one external document, every model tested stated it as established fact instead.",
   "evidence": "synthesis/2 (dataset 267d1954e225c5b6), single_source arm, 10 cases each. REAP 60b 0/10, gemma4-12b 0/10, K2-Horizon-7B 0/10, K2-Horizon-3.7B 0/10 - 40 of 40 cases, every failure classified 'overclaimed'. The system prompt instructed it directly: 'If the only support is a single external document, give the answer but say SINGLE-SOURCE and cite it. Do not refuse, and do not state it as established.' The same models score 10/10 on the conflicting and insufficient arms, so they follow CONFLICT and INSUFFICIENT instructions in the same prompt - it is specifically the provenance-tier condition they do not evaluate.",
   "impact": "The world-view admission rule - single source served with a caveat rather than admitted or quarantined - cannot be implemented by instructing the model. Same shape as the toolcall/1 finding one layer up: an instruction describing a condition the model must DETECT is not acted on, while an instruction naming an action is. Tier is a property the store knows and the model does not.",
   "action": "Caveat at ingest, deterministically, from the provenance tier the store already holds - not by asking the model to notice. Consistent with okf:verifiable-memory-architecture's conclusion on entity grounding: do not try to make the model stop, reject (or qualify) at the boundary."
  },
  {
   "id": "decision-margins-make-judges-runtime-sensitive",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "Judge and tool-use scores are runtime-sensitive because the decisions sit on ~0.4 logprob margins",
   "claim": "Any task whose output is a decision rather than a passage is decided by a first-token margin narrow enough for kernel-level numerical differences to flip it.",
   "evidence": "granite-guardian-4.1-8b, identical request to oMLX and mlx_lm: both produce prompt_tokens=71 and the same first token, so template, bos/eos and prompt construction are identical. mlx_lm's top_logprobs on that token: 'Reject' -0.7500, 'Keep' -1.1250 - a margin of 0.375, about 69% against 32%. The verdict is decided entirely by that one token. oMLX does not return logprobs, so the distributions could not be compared directly, only the argmax.",
   "impact": "Explains why the same weights score 95.0% on mlx_lm and 36.7% on oMLX while generating fluently on both, and why toolcall/1 moved 43 points on REAP from a server change. Screening, abstention and the decision to call a tool are all decision tasks, so all of them are runtime-sensitive by construction. Generation quality is not, which is why nobody notices.",
   "action": "Treat runtime as a controlled variable for any decision-shaped suite, the way dataset and grader already are. Record it in the result identity. Prefer a judge on the runtime it was baselined on."
  },
  {
   "id": "omlx-degrades-what-it-serves",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "oMLX degrades models it has no native kernel for, by up to 58 points",
   "claim": "oMLX ships architecture-specific kernels for a handful of families and falls back to a generic implementation for everything else. The fallback silently loses accuracy.",
   "evidence": "granite-guardian-4.1-8b on screening, 60 cases. Quantisation ladder on mlx_lm: GGUF Q4_K_M via llama.cpp 60/60 (100.0%), MLX bf16 unquantised 59/60 (98.3%), MLX 8-bit 59/60 (98.3%), MLX mixed_3_6 57/60 (95.0%). The SAME mixed_3_6 build via oMLX: 22/60 (36.7%). So conversion and 8-bit cost nothing measurable, mixed_3_6 costs about 3 points, and oMLX costs 58. Ruled out: sampling (byte-identical output at temperature 0 seed 42), KV cache (--no-cache reproduces 36.7% exactly), prompt construction (prompt_tokens=71 on both servers for the same request, yet completions differ), oQ (never invoked on the load path), Bonsai shims (gated to 1-2 bit). The pattern that fits: oMLX ships eleven qwen35_* kernels plus qwen38, gemma4, glm_moe_dsa, deepseek_v4, minimax_m3, laguna, llama4 and others, and NOTHING for granite or qwen3_next. qwen3.8-27b (model_type qwen3_5) scored 60/60 on toolcall/1 under oMLX and 60/60 under rapid-mlx - no degradation. The two models that collapsed are the two with no native kernel path. Third reference added 2026-09-09: the same weights run through BARE mlx_lm in-process, no HTTP and no serving layer, score 55/60 (91.7%, [82-96]) - overlapping the mlx_lm server's 95.0% [86-98] within noise at n=60. oMLX's 36.7% [26-49] overlaps neither, nor the GGUF baseline. Upstream MLX therefore computes these weights correctly and oMLX is the outlier against three independent references.",
   "impact": "oMLX is safe for the families it targets and unsafe for the rest, which is a usability rule rather than a ban. The whole fleet was briefly consolidated onto it because it had the cleanest configuration story. It also does not batch (16.3 aggregate at n=32 against 61-62).",
   "action": "Check omlx/patches for the model's architecture before serving it there. granite and qwen3_next have none. Serve those on mlx_lm or rapid-mlx. Note omlx ships mlx_vlm_qwen4_exp_compat, so Qwen3.8-Flash-Next (qwen4exp) may have an MLX path after all, which mlx_lm does not offer. Reproducible case for upstream: granite-guardian mixed_3_6, screening, 95.0% mlx_lm vs 36.7% oMLX."
  },
  {
   "id": "guardian-mlx-conversion-broke-the-judge",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "Converting the judge to MLX destroyed it, from 100% to 36.7%",
   "claim": "granite-guardian-4.1-8b quantised to MLX mixed_3_6 and served by oMLX loses most of its judging accuracy.",
   "evidence": "screening, 60 cases, 100% coverage. GGUF Q4_K_M via llama.cpp: 60/60 = 100.0%, reproducing the 2026-08-31 baseline exactly. MLX mixed_3_6 (3.8 GB) via oMLX: 22/60 = 36.7%. Not yet separable into quantisation damage versus runtime, because REAP moved 78.3 to 35.0 on a runtime change alone the same day.",
   "impact": "The converted build had already been linked into the serving pool before validation. A judge that silently stops judging corrupts everything downstream that trusts its verdicts.",
   "action": "Judge stays on the GGUF via llama.cpp. Validate any converted model against its own baseline before it serves. Isolate quant from runtime by serving the same MLX build on mlx_lm."
  },
  {
   "id": "runtime-dominates-tool-calling",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "The serving runtime changes tool-calling results more than the model does",
   "claim": "Identical weights and a frozen dataset produce wildly different tool-use scores depending only on which server hosts the model.",
   "evidence": "qwen3-next-80b-reap on toolcall/1 (dataset 10af88c170513155): 47/60 (78.3%) served by rapid-mlx, 21/60 (35.0%) served by oMLX. Per-arm, availability went 12/12 to 0/12 and need 10/12 to 0/12, with 39 cases answering directly instead of calling. Likely mechanism: oMLX applies the model's own chat template, rapid-mlx substitutes its own, and tool rendering is downstream of that.",
   "impact": "Every cross-model comparison taken across different servers is confounded, including the seat assignments derived from them. Quantisation effects measured across runtimes are not separable from the runtime itself.",
   "action": "Hold the runtime fixed in any model or quantisation comparison. A model x runtime 2x2 is required before the existing board can be trusted."
  },
  {
   "id": "if-then-rule-drives-tool-invocation",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "Models call tools when given an if-then rule, not when told a tool exists",
   "claim": "Announcing availability is close to useless; naming the trigger, the action and the displaced default works.",
   "evidence": "toolcall/1, 12 cases per arm. Stating a tool is available: qwen3.6-35b 2/12. Stating 'If a value is missing from the document, you SHOULD call the available tool rather than reporting it absent': 12/12 on the same model. Reproduced through Pydantic AI, which passes the full docstring into the tool description: terse docstring 0/6 on both qwen3.6-35b and REAP, if-then docstring 4/6 on both. qwen3.8-27b is the exception, scoring 12/12 on every arm under two different runtimes.",
   "impact": "Tool descriptions are the portable carrier of the rule across harnesses. A tool that supplies a convention rather than a fact will be skipped unless its description states when to use it.",
   "action": "Write every tool docstring as an if-then rule. Verify per harness that the full description reaches the model, since some send only the summary line."
  },
  {
   "id": "concurrency-n32-is-the-largest-gain",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "Running 32 requests at once is worth 3-4x, and single-stream tok/s is the wrong metric",
   "claim": "Aggregate throughput, not decode rate, governs wall-clock on suites and agent fleets.",
   "evidence": "qwen3.8-27b: 15.3 tok/s at n=1, 62.1 at n=32. REAP: 44.7 to 139.2. Both knee at n=32 and fall off at n=48 (47.5 and 128.9). Wall time is flat from n=8 to n=32, so batching above n=8 is nearly free; the cost is all paid between n=1 and n=8.",
   "impact": "Largest throughput gain found on this box, on a lever that single-stream benchmarking cannot see.",
   "action": "Run bulk extraction at n=32. Note oMLX does not batch (16.3 aggregate at n=32 with the cap raised), so bulk work belongs on mlx_lm or rapid-mlx."
  },
  {
   "id": "rapid-mlx-ignores-chat-templates",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "rapid-mlx drops chat_template_kwargs AND the template's default reasoning block, costing 2x wall clock",
   "claim": "Client-supplied template kwargs never reach the template render, so reasoning_effort cannot be set. The template itself IS applied - the earlier claim that it was ignored was wrong.",
   "evidence": "Same prompt, same weights, both servers. rapid-mlx: 59 prompt_tokens at default, xhigh and low alike - completely flat. mlx_lm: 57 default, 99 xhigh, 87 low, so the template does inject steering text and the kwarg does land. rapid-mlx's 59 sits within two tokens of the UNSTEERED render, not the 99-token xhigh one. Source confirms the intent: vllm_mlx/utils/chat_template.py explicitly passes chat_template_kwargs through, naming reasoning_effort for Qwen3.8, but has an error-driven retry cascade that strips kwargs. Two open upstream issues describe it: #120 (the no-template branch returns plain role:content and drops tools) and #122 (the first retry strips enable_thinking and never restores it). A separate auto-disable heuristic (R12-T2F, 'casual chat request to a thinking-capable model') was investigated and ruled out - it fired once across the substantive batch and the result is identical with or without it. Fair re-measurement 2026-09-09, same curated build on both servers, 3 repeats, identical prompt (the original comparison was confounded: rapid-mlx had the local convert, which has no chat_template.jinja at all, while mlx_lm had the curated build). rapid-mlx 43.6s mean, prompt=49, completion=625, 14.3 t/s. mlx_lm 19.7s mean, prompt=89, completion=279, 14.2 t/s. Decode rate is IDENTICAL - the whole gap is output length. The 40-token prompt difference matches the template's 42-token xhigh steering block, which rapid-mlx does not render. Counterintuitively that block makes output SHORTER ('prioritize correctness, consistency, and clarity'), so the steered model is concise at 279 tokens and the unsteered one rambles to 625.",
   "impact": "CORRECTS two earlier versions of this finding. rapid-mlx does not ignore templates (a template is rendered) and does not force maximum reasoning effort (it renders unsteered). What it does is drop both client kwargs and the template's own default reasoning block. On this model that doubles wall-clock at an identical decode rate. Neither runtime is faster than the other per token.",
   "action": "reasoning_effort is plain text - the template injects a single sentence, and medium is the empty string. It can be sent as a system message to any server. Measured recovery on rapid-mlx: unsteered 625 tokens, injected xhigh 500, against mlx_lm's native 279 - so injection recovers about a third, and a system message is not equivalent to the template rendering the same text in its own position. Use mlx_lm where the steering matters."
  },
  {
   "id": "speculative-decoding-closed-on-mlx",
   "date": "2026-09-09",
   "status": "confirmed",
   "title": "Speculative decoding does not pay on this box except embedded MTP on llama.cpp",
   "claim": "Four separate attempts lose, and the bandwidth explanation from August predicts all of them.",
   "evidence": "DFlash on qwen3.8-27b: 8.7/8.0/9.0 tok/s against a 12.3/8.9/12.3 baseline across creative, structured and code, retested with parsers correct and reasoning captured separately. DDTree refuses to load ('requires tree_budget'; eligible aliases: qwen3.5-9b-8bit only). MTP never engages through rapid-mlx despite the sidecar shipping in the build. Suffix decoding engages and is flat. The DFlash2 drafter is 3.6 GB against a 15 GB target - 24%, in the range the August ratio rule already flagged as too heavy.",
   "impact": "Closes a line of work that consumed several sessions. llama.cpp with embedded MTP remains the only place speculation wins (1.10x chat, 1.35x math).",
   "action": "Stop testing external-drafter speculation on this hardware unless memory bandwidth changes."
  },
  {
   "id": "mlx-ignores-response-format",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "mlx_lm silently ignores response_format",
   "claim": "mlx_lm's OpenAI server accepts response_format and never applies it. Constrained arms become free generation.",
   "evidence": "response_format and json_schema appear 0 times in mlx_lm/server.py (1904 lines). Granite via MLX scored forbid 24/24, permit 24/24, optional 24/24 - perfectly identical arms; via llama.cpp 22/24, 21/24, 22/24.",
   "impact": "15 result files across grounding and nullschema invalidated, spanning 8 MLX runs including qwen3-next-80b and qwen3.8-27b.",
   "action": "Guard added to runner.py: suites sending response_format now refuse to run against a server that fails a sentinel-enum probe."
  },
  {
   "id": "enforcement-is-per-model",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "Schema enforcement is per-model, not per-runtime",
   "claim": "llama.cpp applies grammar for some models and silently drops it for others.",
   "evidence": "Identical strict enum schema: granite-4.2-30b returns {\"v\":\"__sentinel__\"}; qwen3-coder-30b enforced; muse-glimmer-30b returns plain text 'blue'.",
   "impact": "Muse-Glimmer nullschema is unmeasurable on either runtime. Granite's 90.3% is verified valid.",
   "action": "Probe every model, not every runtime. Recorded as a stated gap rather than a number."
  },
  {
   "id": "suites-saturated-at-top",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "The extraction suites can no longer separate the leading models",
   "claim": "At n=72, nullschema cannot distinguish 97% from 89%.",
   "evidence": "qwen3-next-80b 97.2% [90.4,99.2]; granite-4.2-30b 90.3% [81.3,95.2]; gemma4-12b 88.9% [79.6,94.3]. Intervals overlap heavily.",
   "impact": "gemma4-12b at 7GB is statistically indistinguishable from qwen3-next-80b at 40GB on all three extraction suites.",
   "action": "nullschema needs more cases or harder tiers before it can adjudicate the top of the field."
  },
  {
   "id": "fine-tune-regression",
   "date": "2026-09-07",
   "status": "confirmed",
   "title": "The tagger fine-tune scored below its own base model",
   "claim": "tagger-e2b-v1 (LoRA, 270 turns) is worse than gemma4-e2b, its base.",
   "evidence": "Same runtime, same profile, same 240 gold pairs, 100% coverage: tagger 85.8% [81-90] base vs 74.2% [68-79] distill. Non-overlapping.",
   "impact": "Cutover cancelled. gemma4-12b keeps the tagging seat at 93.8/97.9.",
   "action": "Revisit the training data, not the runtime."
  },
  {
   "id": "kv-not-params-decides-fit",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "KV cache decides what fits, and only measurement predicts KV",
   "claim": "Parameter count does not predict memory at context. Neither does the standard KV formula for every architecture.",
   "evidence": "Measured on a context ladder from 8k to 131k, as resident memory: muse-glimmer 1.77 GB of KV, glm-4.7-flash 6.67 GB, granite 30.26 GB. Predicting from 2 x layers x kv_heads x head_dim matched granite (0.250 predicted, 0.236 measured) but overestimated muse-glimmer by 6x and glm-4.7-flash by 7x.",
   "impact": "Granite is the model that runs out of memory at full context, not GLM. Granite reaches 48.62 GB at 131k and breaks the budget. GLM holds the same context in 36.42 GB at 37.2 tok/s.",
   "action": "An earlier version of this finding claimed GLM-4.7-Flash needed 88 GB and could not run here. That was calculated, not measured, and it was wrong by 7x. Head counts in a config do not describe the KV cache for architectures using windowed or compressed attention. Run the ladder."
  },
  {
   "id": "dense-vs-moe-throughput",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "Active parameters, not total, set throughput",
   "claim": "Dense and MoE models of the same total size differ 3-5x in decode speed.",
   "evidence": "Dense 30B: granite 9.0, muse-glimmer 9.6 tok/s. MoE 33-35B: ornith 28.5, qwen3.6-35b 52.4 tok/s. qwen3-next-80b (3B active) 37.7 tok/s.",
   "impact": "A dense 30B is unusable for tool loops but fine for overnight batch extraction.",
   "action": "Judge candidates on active params for speed and total params for memory."
  },
  {
   "id": "dflash-draft-failed",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "Speculative decoding did not help, by either method tested",
   "claim": "Neither a published DFlash draft nor a matched MTP head improved decode speed on this machine. Both made it slower.",
   "evidence": "DFlash on muse-glimmer-30b, target and draft from the same publisher: baseline 5.75 tok/s, speculative 2.65, a 2.2x slowdown, with 11.1% draft acceptance over 1654 generated tokens. MTP on qwen3.6-27b under rapid-mlx, one flag apart: 15.35 tok/s without, 13.5 with, 12% slower.",
   "impact": "Speculative decoding needs roughly 60 to 80 percent acceptance to pay for itself. Every rejected draft token costs a wasted verification pass.",
   "action": "An earlier version of this finding blamed a repository mismatch, because the first DFlash test paired an unsloth target with a meta-models draft. Repeating it with both from one publisher gave 11.1 percent acceptance against 10.8. The pairing was never the cause. The draft does not predict this model."
  },
  {
   "id": "reasoning-hurts-extraction",
   "date": "2026-08-27",
   "status": "confirmed",
   "title": "Reasoning increases misattribution under authority pressure",
   "claim": "Enabling reasoning degrades extraction while leaving detection untouched.",
   "evidence": "authority_lure 100% to 40%. Screening 91.7% both ways with identical failure sets. qwen3-next-80b-thinking nullschema 58.3% vs 97.2% for Instruct.",
   "impact": "The Instruct model holds the extraction seat; Thinking is not a drop-in upgrade.",
   "action": "Reasoning is a dial for agentic work, not a default."
  },
  {
   "id": "specialist-labels-mislead",
   "date": "2026-09-07",
   "status": "confirmed",
   "title": "Specialist labels failed to predict outcomes three times in four",
   "claim": "A model tuned for a task does not reliably win that task's suite.",
   "evidence": "gemma4-12b (general) beats qwen3-next-80b (larger) at tagging by 7.5. Coding models score mid-pack on extraction suites that never measure code.",
   "impact": "Coding models were being ranked on document extraction - a category error, not a model weakness.",
   "action": "Built shapegen and sitebuild to measure code against a contract instead."
  },
  {
   "id": "health-check-must-generate",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "HTTP 200 does not mean a model is loaded",
   "claim": "Readiness probes that check an endpoint rather than generation produce silent garbage runs.",
   "evidence": "mlx_lm returned 200 on /v1/models while the laguna architecture was unsupported and every request failed; the run hung for 2 hours. Earlier, curl && break accepted a 503 and 317 cases ran against a loading server.",
   "impact": "Two multi-hour runs lost.",
   "action": "Readiness now requires a successful completion, with >=512 max_tokens so reasoning models are not starved."
  },
  {
   "id": "three-bottlenecks",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "Three different bottlenecks, depending on how the model is served",
   "claim": "Dense models are bound by memory bandwidth, mixture of experts models by active parameters, and streamed models by SSD throughput times cache hit rate. The same machine gives very different ceilings depending on which applies.",
   "evidence": "Measured on one M1 Max with 64 GB: memory bandwidth about 400 GB/s, sequential SSD read 6.34 GiB/s, a ratio of roughly 63 to 1. Dense 30B models run 9 to 11 tok/s. MoE models with 3B active run 37 to 52 tok/s at similar total size. A published SSD-streamed build measured 5.33 tok/s on a 12.47 GiB/s disk, which scales to about 2.7 tok/s here.",
   "impact": "Streaming buys capacity up to the memory ceiling, not past it. An M64-layout build runs 79.1 GB of file at roughly 43 GB resident and works. A ternary streamed build needing 84.1 GiB peak does not, however little of it is resident at once.",
   "action": "Judge a candidate by which bottleneck applies to it, not by parameter count. Slow is acceptable for unattended batch work; exceeding the memory ceiling is not."
  },
  {
   "id": "reap-preserves-abstention",
   "date": "2026-09-08",
   "status": "confirmed",
   "title": "Expert pruning cost this model nothing measurable",
   "claim": "REAP removes a quarter of Qwen3-Next-80B's parameters and the extraction behaviour is preserved, including the abstention that won it the seat.",
   "evidence": "Same suites, same grader, same profile, schema enforcement verified. nullschema 70/72 for both, identical failures. grounding 111/125 full against 109/125 pruned. screening 55/60 against 59/60. Throughput 37.7 against 41.4 tok/s. Weights 40 GB against 30.4 GB, 80B params against 60B.",
   "impact": "The pruned build can hold the extraction seat, freeing 10 GB and running 10 percent faster. On a 64 GB machine that is the difference between one large resident model and two.",
   "action": "Prune before buying memory. This was a 30 GB download that returned more usable capacity than any deletion made today."
  },
  {
   "id": "byte-alignment-governs-decode",
   "date": "2026-09-14",
   "status": "revised",
   "title": "Byte alignment, not bit count, governs decode speed",
   "claim": "REVISED -- scoped to DENSE models; unvalidated on MoE. Quantisations whose bit width divides a byte evenly (8, 4, 2) sustain far more of peak memory bandwidth than those that straddle byte boundaries (5, 6). A smaller file can be slower than a larger one.",
   "evidence": "Qwen3.5-9B, same weights requantised four ways, repeated on two prompts with under 0.3 percent variance. Q4_0 45.4 tok/s at 60 percent of 400 GB/s; Q6_K 32.1 at 59 percent; Q8_0 31.4 at 75 percent; Q5_0 30.5 at 48 percent, the worst despite being smaller than Q6_K and Q8_0. Stock Q4_K_M 32.8 at 47 percent. Independently reproduced by llama-bench: Q4_0 44.59 +/- 0.01 against Q4_K_M 31.45 +/- 0.03, a 1.42x gap. Same pattern on the 27B: Q4_0 15.8 tok/s at 62 percent beats IQ3_S 11.5 at 35 percent while being 30 percent larger. SCOPE (2026-09-14): every model in this evidence set whose architecture is recorded is DENSE -- granite-guardian-4.1-8b and qwen3.8-27b confirmed from their GGUF headers; k2-36b is no longer on disk and its architecture is not recorded anywhere. The ratio above is therefore a dense-model result and was stated without scope. Suggestive counter-observation, NOT a measurement: two 35B-A3B MoEs serving side by side gave agents-a1 (Q4_K_M) 49.1 t/s against ornith-1.5 (Q4_0) 53.4 t/s, a ratio of 1.10x. Those are DIFFERENT WEIGHTS, so it is not a quantisation comparison and cannot confirm anything. No MoE model exists on this disk in two formats, so the clean same-weights test has not been run.",
   "impact": "Predicts which formats are worth testing without testing them. Five of six fleet seats currently run K-quants and are leaving roughly 1.4x on the table. Q5_0 and Q6_0 are ruled out as a sweet spot. The mechanism predicts the gap: byte misalignment is paid per byte actually read, and an MoE reads only its active experts (8 of 256 here, ~3B of 35B params), so the absolute penalty should shrink roughly in proportion to sparsity. Predicted, not measured.",
   "action": "Serve Q4_0 wherever a build exists. Do not chase sub-4-bit for speed; the unpacking cost grows faster than the byte saving. Do NOT apply this ratio when sizing or selecting an MoE seat -- it is unvalidated there. To close the gap, requantise one MoE to a second format (agents-a1 Q4_K_M -> Q4_0 is the cheapest path; decode speed depends on layout, not on requantisation quality loss) and run llama-bench on both. RESOLVED 2026-09-15: the MoE question is now measured, and the answer differs by format family -- see iq-codebook-penalty-transfers-to-moe-kquant-does-not. K-quant penalty falls to 1.10x on a sparse MoE; IQ codebook penalty RISES to ~2.1x. Use that finding, not this one, when sizing an MoE seat."
  },
  {
   "id": "speculative-decoding-net-loss",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Speculative decoding is a net loss on this hardware, in every arm tested",
   "claim": "Draft acceptance was healthy and throughput still fell. Eighteen arms across two models, three workloads, four draft depths and two mechanisms produced no win.",
   "evidence": "Qwen3.8-27B dense IQ3_S with in-file MTP head: baseline 11.5 tok/s, best speculative arm 9.1 (0.79x), monotonically worse with depth. gemma-4-26B-A4B with a 0.53 GB EAGLE-3 head: baseline 68.5, best arm 56.4 (0.83x). Acceptance was 2.0 to 2.9 tokens per target forward throughout, so drafts were good; the verify path costs roughly 3x a plain forward. Measured with llama-cli; llama-server batching untested.",
   "impact": "Closes a line of work that had already consumed DFlash, EAGLE-3 and MTP investigations. Explains three previously separate puzzles as one: DFlash failing on prose, k2-36b at 0.51x, and EAGLE acceptance disappointing.",
   "action": "Stop evaluating speculative decoding on this box unless llama-server batching is shown to differ."
  },
  {
   "id": "ternary-format-fast-weights-hard",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "TQ2_0 is fast and lossless to pack, but ternary weights cannot be made post hoc",
   "claim": "llama.cpp TQ1_0/TQ2_0 are containers for already-ternary weights, not quantisation algorithms. The packing is exact, and naive post-training ternarisation destroys the model.",
   "evidence": "Read the kernel: quantize_row_tq2_0_ref sets block scale d = amax then rounds x/amax, so weights already of form {-s,0,+s} round-trip exactly, while normal weights put roughly 87 percent of values at zero. Built the pipeline end to end with stock tools: F16 to BitNet absmean ternarisation (31.4 percent zeros) to TQ2_0, 3.0 GB at 2.85 BPW, 50.5 tok/s, beating Q4_0 45.4. Perplexity went from 8.78 to 51702 on the same corpus. A properly trained ternary model, Bonsai-27B TQ2_0, scores 9.67 on that corpus at 40 percent of peak bandwidth.",
   "impact": "No fork is needed to run or pack ternary, contrary to earlier assumption; only TAARDIS rotated-basis builds need one. Making our own useful ternary requires Hadamard rotation plus learned error correction, which is a research project, not a conversion step.",
   "action": "Use ternary models other people trained. Do not build a ternarisation pipeline unless it becomes a research line in its own right."
  },
  {
   "id": "eagle-vendor-acceptance-inflated",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Published EAGLE-3 acceptance rates did not survive either check",
   "claim": "Documentation claims 75 to 85 percent draft acceptance for EAGLE heads against 55 to 65 for separate draft models. Neither figure held.",
   "evidence": "K2-Horizon-36B EAGLE3 head ships its own val_metrics.json showing full_acc 0.494, 0.230, 0.108 across positions and cond_acc flat near 0.47, so 49 percent per step. Measured directly on gemma-4-26B-A4B: 31.2 percent per-token acceptance over 32 cycles. Squish own EAGLEHeadRunner implements should_fallback() dropping to NGRAM below a threshold, which presumes poor acceptance is normal.",
   "impact": "Speculation cost models built on vendor acceptance figures will be wrong by roughly a factor of two.",
   "action": "Treat spec-decoding README numbers as marketing. Measure acceptance before modelling throughput."
  },
  {
   "id": "qwen3-next-80b-q40-throughput",
   "date": "2026-09-11",
   "status": "confirmed",
   "title": "Qwen3-Next-80B at Q4_0 runs 46.7 tok/s and is workload-flat",
   "claim": "The 80B in Q4_0 is the fastest large model on the board and shows almost no variation across workload type.",
   "evidence": "45.3 GB file, three workloads on one build (c32d1dabe): json 46.6, prose 46.8, code 46.6 tok/s; prompt ingestion 127 to 157 tok/s. The previously recorded 41.4 tok/s IQ4_XS figure came from a different build and IQ4_XS is no longer on disk, so no speedup is claimed.",
   "impact": "An 80B is viable as an interactive seat on this box, not just a batch model.",
   "action": "If a Q4_0 versus IQ4_XS comparison is wanted, re-pull IQ4_XS and measure on the same build."
  },
  {
   "id": "misattribution-is-an-admission-problem",
   "date": "2026-09-13",
   "status": "confirmed",
   "title": "Long-context recall does not degrade with depth; misattribution does the damage, and a string comparison catches all of it",
   "claim": "The failure mode in long-context retrieval is not forgetting, it is returning a similar fact about a different subject. That error is mechanically detectable without a model, because the model cites the correct source for the wrong subject. Checking that the cited span mentions the queried subject moves needle/1 from 58.2% to 98.2% and costs no true recall.",
   "evidence": "needle/1, dataset 5c5199b9eee8a595, ornith-1.5-9b Q4_0, 110 cases across 8k and 32k of real turnpair filler at five depths. present 50/50 (10/10 at each of 10/30/50/70/90 percent), absent 10/10, distractor 4/50 (8.0%). Neither depth nor context length moved any arm. All 46 distractor failures cited a turn id, and cited the RIGHT one - the planted decoy - so no fabrication was involved. Applying a subject-grounding admission check (normalise both sides, require every content word of the subject to appear in the cited span) yields 108/110 (98.2%), recovering all 46. Strictness sweep at 0.25/0.50/0.75/1.00 recovers 2/20/38/46 with the rejection count constant at 2, so there is no precision-recall trade in this data. The 2 are cases where the model returned the correct value citing unrelated filler (turn-0060); needle's grader scores those as retrievals by design, so the filter catches two the grader admits - consistent with epistemics' rule that a right answer with a fabricated citation is not a pass. This is verifiable-memory-architecture section 3 applied to values rather than entities. Not caught: CONFLICT, where two contradictory facts are both correctly cited and both admissible. Tool: benchmarks/tools/admit.py"
  },
  {
   "id": "kv-quantisation-degrades-citation-not-recall",
   "date": "2026-09-13",
   "status": "confirmed",
   "title": "Quantised KV degrades which source a model cites, not what it answers - and the raw grader cannot see it",
   "claim": "q4_0 KV cache costs nothing measurable in answer accuracy and a real, one-directional amount in citation accuracy. Whether quantised KV is 'free' therefore depends entirely on whether the consuming layer admits on citations. The same measurement supports opposite policies.",
   "evidence": "needle/1, dataset 5c5199b9eee8a595, ornith-1.5-9b Q4_0, 110 cases paired across f16 and q4_0 KV at 8k and 32k over five depths, identical weights, sequential runs, no contention. Graded two ways. Raw grader: f16 64/110, q4_0 61/110, 11 discordant pairs split 7 to 4, McNemar exact p=0.549 - no detectable difference. With citation-grounded admission (tools/admit.py): f16 108/110, q4_0 99/110, 9 discordant pairs split 9 to 0, p=0.004. Every divergence favours f16. Median latency identical at 24.4s vs 24.5s, so the quantisation buys memory and not speed: 25.7 GB to ~11 GB at 131072 context on this model, 128.0 to 32.0 KB per token.",
   "impact": "Reverses the reading this run was commissioned to produce. The first pass concluded 'quantised KV is free at depth' and recommended applying it to the driver seat for 14.7 GB. That was true of the measurement and false of the policy. needle's grader scores a retrieval correct even when the citation is wrong, which is a deliberate choice to separate recall from citation - and it is precisely the choice that hides this effect. Epistemics already states the rule: a right answer with a fabricated citation is not a pass.",
   "action": "Driver seat stays on f16 KV: it feeds FreshLoops and the graph, where admission is citation-grounded, so citation accuracy is load-bearing. Coding seats use q4_0 KV: nothing admits their output on citations, so the memory is bought in a currency they do not spend. kv-q40-driver.sh now refuses without --accept-citation-cost and prints the trade. Not generalised beyond one model and one suite; architectures with windowed or compressed attention were not tested."
  },
  {
   "id": "iq-codebook-slower-than-larger-byte-aligned",
   "date": "2026-09-13",
   "status": "confirmed",
   "title": "A 23% smaller IQ3_S build decodes 27% slower than Q4_0 on the same weights - the codebook costs more than the bytes it saves",
   "claim": "IQ codebook quantisation loses more to lookup cost than it gains from a smaller footprint. Choosing a quant by file size is choosing the wrong variable; byte alignment beats bit count across quant families, not just within the K-quant series.",
   "evidence": "Qwen3.8-27B, identical weights (866 tensors, 27.32B params), llama-bench c32d1dabe (b10893) on a quiet box with all seats stopped, -p 512 -n 128 -r 3. Q4_0 (file_type=2, 14.63 GiB): prefill 118.48 +/- 0.05 t/s, decode 15.26 +/- 0.01. GSQ-RCO IQ3_S (file_type=26, 3.4375 bpw, 11.28 GiB): prefill 100.87 +/- 0.07, decode 11.21 +/- 0.03. So 23% smaller yields 15% slower prefill and 27% slower decode. Effective decode bandwidth: Q4_0 moves 14.63 GiB at 15.26 t/s = ~223 GB/s (56% of the M1 Max's 400 GB/s); IQ3_S moves 11.28 GiB at 11.21 t/s = ~126 GB/s (32% of peak) - inside the 35-48% band byte-alignment-governs-decode records for IQ formats. Error bars across three reps are 0.01-0.07 t/s, so the gap is not noise.",
   "impact": "The GSQ build was adopted on a vendor claim of parity with unquantised Qwen3.8 and is the likely cause of the standing impression that the 27B dense model is 'excellent but slow'. On the capability side the two builds are already measured as equal on toolcall (100.0), toolscale (100.0), synthesis (83.3) and worldview (77.5), and GSQ is 13.9 points WORSE on nullschema (62.5 vs 76.4). The smaller build is therefore slower and weaker, winning only on disk - of which there is 258 GB free.",
   "action": "Serve Qwen3.8-27B from the Q4_0 build, already on disk. Do not select a quantisation by file size. Extends byte-alignment-governs-decode from the K-quant series to IQ codebook formats: the rule is cross-family."
  },
  {
   "id": "iq-codebook-penalty-transfers-to-moe-kquant-does-not",
   "date": "2026-09-15",
   "status": "confirmed",
   "title": "Sparsity rescues a model from K-quant misalignment and not from IQ codebook lookup",
   "claim": "The dense byte-alignment penalty splits on MoE. A K-quant costs 1.42x on a dense model and only 1.10x on a sparse MoE. An IQ codebook costs 1.37x dense and roughly 2.1x on the same MoE. Misalignment is a memory-traffic cost and sparsity cuts traffic; codebook lookup is an arithmetic cost paid per ACTIVE weight, which sparsity does not reduce.",
   "evidence": "Same weights, two builds, one box. BTL-4 (35B total, 2.1B active, 256 experts, 8 routed) served on llama.cpp b10893: IQ2_XXS 23.6 tok/s decode and 38-71 tok/s prefill; Q4_K_M 48.5 tok/s decode and 82-144 tok/s prefill, three runs each within 0.2 percent. That is 2.06x from the quantisation alone. Cross-checked against ornith-35b-a3b Q4_0 (3.0B active) at 53.4 tok/s: from active parameters alone BTL-4 should reach ~76 tok/s, so IQ2_XXS was costing ~3.2x and Q4_K_M ~1.6x. The K-quant half comes from agents-a1-35b Q4_K_M 49.1 against ornith Q4_0 53.4, both 256-expert MoE.",
   "impact": "Corrects the scope note added to byte-alignment-governs-decode on 2026-09-14, which said only that the dense ratio was unvalidated on MoE. It is now measured and the answer differs by format family. Choosing an IQ build for an MoE seat costs roughly twice what the dense finding predicts; choosing a K-quant costs almost nothing.",
   "action": "Never serve an MoE from an IQ codebook build if any byte-aligned or K-quant build exists. Q4_K_M is an acceptable MoE seat despite the dense 1.42x. Re-read fleet_audit output with this split in mind."
  },
  {
   "id": "iq2-quantisation-costs-capability-not-just-speed",
   "date": "2026-09-15",
   "status": "confirmed",
   "title": "IQ2_XXS took 27 points off world knowledge and 17 off tool invocation on identical weights",
   "claim": "An aggressive IQ quantisation degrades capability far beyond its throughput cost, and the damage is uneven across suites rather than a uniform decline.",
   "evidence": "BTL-4, same weights, 14 suites, one harness, temperature 0. IQ2_XXS to Q4_K_M: toolcall 76.7 to 93.3 (+16.7), nullschema 36.1 to 44.4 (+8.3), contract 64.6 to 70.8 (+6.2), synthesis 80.0 to 83.3 (+3.3). Not everything improved: worldview 62.5 to 55.0 (-7.5) and tagger_nullable 96.2 to 94.2 (-2.0). epistemics held at 100.0 and enrichment at ~98.7 in both.",
   "impact": "A model screened on an IQ build can be rejected for a capability it actually has. BTL-4 on IQ2_XXS looked like a mediocre tool-caller at 76.7 percent; on Q4_K_M it is 93.3 percent and beats the incumbent driver. The uneven direction also means a single aggregate score hides the effect.",
   "action": "Screen models on the best available build, not the smallest. Where only an IQ build is published, record the quantisation next to the score and treat the result as a floor."
  },
  {
   "id": "abstention-wall-first-model-above-forty-percent",
   "date": "2026-09-15",
   "status": "revised",
   "title": "Abstention ranges 18-61 percent across the fleet, and declining is not the same skill as noticing a correction",
   "claim": "CORRECTED 2026-09-15, twice in one day. The original claim -- that nullschema sits between 18 and 44 percent across every model measured, and that BTL-4 Q4_K_M at 44.4 was the first above 40 -- was FACTUALLY WRONG. qwen3.6-35b-a3b scored 61.1 percent (n=72) and was already in the results when the finding was written; it was omitted from the comparison set. The real range is 18.1 to 61.1. Abstention is still a fleet weakness, but it is not a wall at 44, and the gap between best and worst is far wider than stated.",
   "evidence": "nullschema, 72 cases each: qwen3.6-35b-a3b 61.1, BTL-4 Q4_K_M 44.4, BTL-4 IQ2_XXS 36.1, ornith-35b-a3b-q40 36.1, MiniCPM5-2B 31.9, ternary-bonsai-27b 27.8, bonsai-8b-1bit 19.4, bonsai-27b-1bit 18.1, qwen3.8-9b-distill 18.1. nullschema, 72 cases each: BTL-4 Q4_K_M 44.4, BTL-4 IQ2_XXS 36.1, ornith-35b-a3b-q40 36.1, ternary-bonsai-27b 27.8, bonsai-27b-1bit 18.1, bonsai-8b-1bit 19.4, qwen3.8-9b-distill 18.1. Best-in-class before this session was 36.1. CRITICAL QUALIFIER: worldview/2 breaks abstention into arms, and the ordering inverts. On the `superseded` arm -- a fact whose correction is also on record -- BTL-4 Q4_K_M scores 0/8 and ternary-bonsai-27b 0/8, while ornith-35b-a3b-q40 scores 8/8. BTL-4's failure mode is recorded as right_value_not_flagged (8) and overclaimed (7): it retrieves the stale value and states it without qualification. So the best abstainer on absent fields is among the worst at spotting a superseded one. `single_source` is worse still and fleet-wide: 0/8, 0/8, 2/8 -- nothing reliably flags uncorroborated evidence, consistent with models-do-not-detect-conditions.",
   "impact": "The omission mattered because the finding was used to argue BTL-4 belonged in front of the graph. On absent-field abstention qwen3.6-35b is materially better, and it also leads grounding at 82.4 (n=125) against BTL-4's 76.8 and ornith's 76.0. Directly constrains the FreshMap rule that PII never hits the graph. No model in the fleet can be trusted to decline on its own, so the grounded-admission filter is not an optimisation but a requirement. It also explains why a self-reported confidence field is the wrong design: a model that fabricates in three cases out of four is not a reliable narrator of its own certainty. And it means a single nullschema figure must not be read as 'this model knows when not to answer'. A model can lead on absent-field abstention and still assert a corrected fact as current -- which is the more dangerous failure for a store that keeps its own corrections.",
   "action": "Build the comparison set from every label in benchmarks/results, not from the models under discussion. This finding was written from a four-model table while nine models had scores. Keep admit.py in front of anything that writes to the graph regardless of which model is serving. Use nullschema as the gate for any new model considered for an extraction role, and treat anything below 50 percent as requiring external grounding. Screen an extraction candidate on nullschema AND on worldview/2's superseded and single_source arms, not nullschema alone. Where a role depends on using the latest version of a record, ornith-35b-a3b is currently the only model measured that does it reliably."
  },
  {
   "id": "runtime-does-not-change-capability-on-dense-models",
   "date": "2026-09-15",
   "status": "confirmed",
   "title": "MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites",
   "claim": "Extends the 2026-09-11 MoE result to a dense model. Serving the same weights through llama.cpp and through mlx_lm produces the same scores; runtime choice is an operational decision about speed and memory, not a capability one.",
   "evidence": "Ternary-Bonsai-27B, GGUF PQ2_0 on the prism llama.cpp fork against MLX 2-bit on mlx_lm, 12 paired suites. Eight exact ties including toolcall, tools, toolscale, screening, synthesis, worldview and contract. The two 240-case suites agree to within half a point: tagger 96.2 vs 96.5, tagger_nullable 97.9 vs 97.4. Only epistemics (+4.2) and grounding (+2.4) move, both toward MLX, and both reproduced independently on a second machine. Mean delta +0.55. chainwalk excluded: n=2.",
   "impact": "Removes runtime as a variable when comparing models, provided the build string is recorded. It also means the prism MLX fork buys nothing in capability over the llama.cpp fork, so the choice between them rests on throughput and on which is less work to maintain.",
   "action": "Continue recording the server build string per run; two llama.cpp builds were compared as one model earlier in this session and only configuration_id caught it. Do not treat an MLX and a GGUF score as different models."
  },
  {
   "id": "expert-prefetch-helps-decode-not-prefill",
   "date": "2026-09-16",
   "status": "confirmed",
   "title": "A trained expert prerouter buys 17% on decode and nothing on prefill",
   "claim": "In an SSD-streamed MoE, predicting which experts the router will pick pays only during decode. Prefill sees no benefit at all, because a long prompt touches nearly every expert anyway -- there is nothing left to predict.",
   "evidence": "Edge0-35B-A3B (Qwen3.6-35B-A3B base, int4 + recover-LoRA, 256 experts, top_k=4), same checkpoint, same box, prerouter toggled with --no-prerouter. Decode, 300-token generations, warm runs: 14.65 tok/s ON against 12.55 OFF, +16.7 percent. Prefill on a 3,725-token prompt: 282.4 ON against 284.9 OFF, and 261.8 against 265.6 -- a 1 percent deficit, inside noise, with OFF marginally ahead both times.",
   "impact": "Decode issues 160 individually-stalling random expert reads per token at 40 layers x top_k 4, and each stall is latency the prefetch can hide. Prefill processes thousands of tokens at once, so the union of experts approaches all 256 and the read is effectively bulk. This predicts the shape of any expert-streaming engine: latency-hiding helps the token-at-a-time path and not the batch path.",
   "action": "When evaluating a streaming MoE engine, measure decode and prefill separately. A single aggregate throughput figure hides which half the engineering actually affects."
  },
  {
   "id": "prefetch-gain-shrinks-as-storage-gets-faster",
   "date": "2026-09-16",
   "status": "confirmed",
   "title": "The faster the SSD, the less an expert prerouter is worth -- which inverts who should want it",
   "claim": "Prefetching expert weights pays in proportion to the I/O stall it hides. On fast storage there is little stall, so the gain collapses. The machines that benefit most from streaming architectures are the cheap ones, not the good ones.",
   "evidence": "Edge0 publish 'up to +59 percent decode; the gain grows with storage latency, model size and routed width K'. Measured on cortext -- 20.2 GB/s NVMe, 1.0M IOPS at 4K, the fastest tier in this programme's own storage table -- the same toggle yields +16.7 percent. The vendor claim and this measurement are consistent rather than contradictory: cortext has the least stall available to hide.",
   "impact": "Reverses the intuition for fleet planning. A 16 GB Mac mini with slow internal storage would see a LARGER prerouter gain than a 64 GB Mac Studio with a 20 GB/s NVMe -- from a lower base, but the architecture is aimed at exactly the hardware that a RAM-resident model cannot fit. It also means any streaming benchmark is uninterpretable without the host's measured disk throughput printed beside it.",
   "action": "Record measured storage throughput alongside every streaming-engine result. Do not port a streaming speedup figure between machines."
  },
  {
   "id": "streamed-moe-footprint-is-committed-not-cached",
   "date": "2026-09-16",
   "status": "confirmed",
   "title": "An mmapped 19.6 GB checkpoint charges 5.3 GB, plateaus in 250 tokens, and reads as 20 GB if you look at the wrong column",
   "claim": "For an SSD-streamed MoE the number that matters is phys_footprint, not PhysMem used and not RSS. Edge0-35B-A3B commits 5.3 GB and stops; the rest is reclaimable page cache that only accumulates because RAM is idle.",
   "evidence": "Measured with macOS `footprint -p`, which excludes clean file-backed pages. At load: 407 MB. After 250 tokens: 5,269 MB. Across the next 2,250 tokens it moved 8 MB -- 5,343 / 5,349 / 5,342 / 5,346 / 5,347 / 5,355 / 5,351 MB. Flat. Meanwhile file-backed grew 16.9 to 19.9 GB, the whole checkpoint, and RSS read 11.84 GB. PhysMem 'used' climbed 17 GB across three generations.",
   "impact": "Three different readings of the same process support three different conclusions. The vendor claims 2.9 GiB (optimistic by ~1.8x), a naive reading of PhysMem says 20 GB (wrong -- that is evictable cache on an idle 64 GB box), and the truth is 5.3 GB committed. Community reports that it runs on a 16 GB Mac mini are correct. This programme initially reported the 20 GB figure, repeating the exact error its own bench-coresidency runbook warns about: 'co-residency measured by available memory is wrong'.",
   "action": "Use `footprint -p <pid>` for any mmap-backed model. RSS and PhysMem both overstate it, and the overstatement scales with how much free RAM the host happens to have."
  },
  {
   "id": "compression-tax-confirmed-by-size-crossed-bitwidth",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "A controlled 2x2 confirms it: the 27B loses 7.3 points to 1-bit, the 8B loses nothing",
   "claim": "Aggressive quantisation costs capability in proportion to the capability present. Crossing model size with bit width on one family, the larger model degrades and the smaller one does not measurably change -- because it had nothing to lose on the suites where the larger one fell.",
   "evidence": "prism-ml Bonsai, two sizes x two bit widths, 12 paired suites, one harness, one box. 27B ternary (PQ2_0, 2.13 bpw) to 27B 1-bit (Q1_0, 1.13 bpw): mean -7.3 points. 8B ternary to 8B 1-bit: mean +1.3, inside noise. Per suite the pattern is explicit -- contract 75.0 to 25.0 on the 27B (-50.0) against 22.9 to 20.8 on the 8B (-2.1); synthesis 66.7 to 35.0 (-31.7) against 60.0 to 65.0 (+5.0). toolcall is 100.0 at both bit widths on the 27B and 0.0 at both on the 8B: a capability that was never present cannot be compressed away.",
   "impact": "Upgrades quantisation-tax-scales-with-capability from an inference across different models to a controlled result within one family. It also means benchmark-average degradation figures published by vendors are close to meaningless without knowing the baseline capability per axis -- a model that scores badly everywhere will appear robust to compression.",
   "action": "When assessing a quantisation, compare per-suite against the SAME model at higher precision, never across models. Expect the largest drops on the suites where the full-precision model scores highest, and treat a flat degradation profile as evidence of a weak baseline rather than a good quantisation."
  },
  {
   "id": "a-broken-model-and-an-incapable-one-score-the-same",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "A destroyed model and an incapable one produce the same zero, so the harness now refuses to run without a coherence check",
   "claim": "A quantisation that destroys a model does not announce itself in any score. It produces 0% on tool calling, 0% on chainwalk, near-zero everywhere, and those numbers are indistinguishable in a results file from a model that is merely weak at those tasks. The distinguishing evidence is only ever in the generated text, which nobody reads once a run has produced a tidy table. A closed-set precondition separates the two cases in under a second: ask questions whose answer space has two members, then check that the answer is IN the set, that it is the RIGHT member, and that the token distribution is peaked rather than flat.",
   "evidence": "PTQ1_0 MiniCPM5-2B (1.75 bpw blanket ternary, 772 MB) ran at 105.7 t/s and scored toolcall 0.0%, which was filed and quoted before anyone read its output: 'YHYHYHYHYHRqRqRqRq...' and, on an earlier sample, '\u4e0d\u8d77 priorichter\u4e0d\u8d77vern'. Eight closed-set probes against those same weights: membership 0/8, accuracy 0/8, mean top-token probability 0.08. The same probes against Ornith-35B-A3B Q8_0: membership 8/8, accuracy 8/8, top-token probability 0.94, and bit-identical across three consecutive runs including the confidence to three decimals. Healthy-path cost 0.9-1.3s wall clock including interpreter start. A merely-weak model answering one probe wrong still passes at 7/8.",
   "impact": "Four false zeros were quoted in one week for this reason. The gate now runs in harness/coherence.py before any case executes and raises SystemExit, so no file is written at all: an attempted toolcall run against the destroyed weights produced no results file. Notably the non-Latin-script heuristic scored 0% on the live failure because that run emitted Latin gibberish rather than CJK, so script detection alone would have missed it entirely; membership and logprob peakedness are the gates that actually fire. Three independent gates exist because any single heuristic has a blind spot.",
   "action": "Run the coherence gate before trusting any score from a newly quantised, newly converted or newly downloaded artefact, and treat a zero from an ungated run as unproven. Never report confidence as a sole failure and never report a thinking-budget overrun as bad weights: the first version of this check sent 24 tokens with thinking enabled and scored healthy Ornith-Q8 at membership 0%, because the entire budget went to reasoning_content and content came back empty. A precondition that fails good models gets switched off, and then the thing it guards against happens anyway. --no-coherence exists for base/completion models and is recorded in the manifest. And treat a pass as a floor, never a quality signal: the partial-ternary Ornith passed this gate at 7/8 with confidence 0.64 while carrying 2.9x the perplexity of its Q8 reference. The gate separates rubble from usable; it does not rank the usable."
  },
  {
   "id": "prism-ternary-is-a-container-not-a-quantiser",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "PQ2_0 and PTQ1_0 cannot post-quantise a trained model, and selectivity does not rescue them",
   "claim": "prism's PQ2_0 (2.13 bpw) and PTQ1_0 (1.75 bpw) work as containers for a model that was TRAINED ternary, and fail as post-training quantisers for one that was not. Restricting them to routed-expert gate/up projections, giving them a 99.6%-coverage imatrix, and backing the bit width off from 1.75 to 2.13 all fail to change that. There is no path to shrinking Ornith below its Q4_0 seat with these formats.",
   "evidence": "Ornith-35B-A3B (hybrid MoE: 41 layers, 30 SSM + 11 attention, 92.8% of weight in routed experts), converted HF to Q8_0 GGUF, imatrix over 220 chunks of generic corpus, held-out perplexity on 120 chunks of unseen prose. Q8_0 reference 35.2 GB PPL 6.83. All-experts Q4_0 control 20.3 GB PPL 7.15 (+4.7%), coherence 8/8 at confidence 0.99. Ternary on 21 of 41 layers 16.2 GB PPL 19.70 (+188%). PQ2_0 on all gate/up 14.7 GB PPL 43.47 (+536%). PTQ1_0 and PQ2_0 variants both scored exactly 50% on eight closed-set probes -- chance -- and failed the SAME three probes identically despite a 22% bit-width difference, which is systematic damage rather than quantisation noise. The q4_0 control isolates it: same requantise path, same imatrix, same recipe mechanism, same hybrid architecture, and it passes clean.",
   "impact": "Closes the 'shrink the fleet safely' thesis for this route. Ornith's floor is the Q4_0-class seat at 20.3 GB, and the 36% saving that a 13 GB mixed-ternary build appeared to offer is not available. It also separates two things that looked like one result: prism-ml Bonsai scoring toolcall 100.0 at PQ2_0 is a NATIVELY ternary model in a ternary container, and says nothing about ternarising anything else. Same distinction already recorded for TQ1_0/TQ2_0 in ternary-format-fast-weights-hard; it now extends to the prism formats, which were the ones with Metal kernels and therefore the only ones that looked viable on Apple Silicon.",
   "action": "Do not post-quantise to PQ2_0/PTQ1_0. Treat a ternary format as usable only where the weights were trained for it. When testing any quantisation, always run an all-Q4_0 control through the identical pipeline first -- it is what distinguishes a broken format from a broken toolchain, and it took one minute."
  },
  {
   "id": "exl3-matches-q8-on-capability-and-loses-3x-on-speed",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "EXL3/QTIP at 4 bpw costs nothing measurable in capability and three quarters of the throughput",
   "claim": "EXL3 at 4.01 bpw on the linear layers scores within noise of the same model at GGUF Q8_0 across eight suites measured the same day on one box, while running at roughly a third of the speed. The quantisation is close to free; the runtime is not.",
   "evidence": "MiniCPM5-2B, nine suites through the harness against PonyExl3 on Apple Silicon, paired against GGUF Q8_0 re-measured today at -np 1. Mean delta -1.6 points over eight paired suites (epistemics +2.1, screening +10.0, tagger -0.4, tagger_nullable -6.7, enrichment +1.9, grounding +4.0, synthesis -3.3, worldview -20.0). Throughput 19.8-23.0 t/s against 71.2-77.2 t/s for llama.cpp Q8_0 on the same machine, a consistent 3.2-3.7x gap. needle 85.5% at n=110 with prompts to 33.6k tokens is EXL3-only; there is no GGUF pairing for it. shapegen refused to run: its cases have drifted from the dataset lock without a SUITE_VERSION bump.",
   "impact": "Confirms EXL3/QTIP is a real option on Apple Silicon rather than a CUDA-only curiosity, and extends runtime-does-not-change-capability-on-dense-models to a third runtime. The worldview -20.0 is the one result that does not fit and is the obvious thing to chase next. In practice the 3x throughput gap decides it: for a 2B model the 35% footprint saving does not pay for itself unless memory is the binding constraint.",
   "action": "Use EXL3 where footprint is the constraint and latency is not. Do not quote the earlier 27.4 t/s EXL3 figure -- it was measured through a server that never stopped generating."
  },
  {
   "id": "the-provenance-stamp-did-not-identify-the-weights",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "One baseline was 25 points off and nothing in the stamp could explain it; the other six reproduced exactly",
   "claim": "The recorded minicpm5-2b tagger baseline of 51.7% could not be reproduced. Re-measured today it scores 76.9%, with identical dataset_id, grader_id, suite_version, profile, llama.cpp build, slot count and weights file on disk. Every hypothesis was testable and every one was wrong; what remained was that the stamp does not capture enough to tell the two runs apart.",
   "evidence": "tagger, MiniCPM5-2B Q8_0, build b10893-c32d1dabe throughout. Board baseline 51.7%. Today: --jinja -np 1 76.7%, --jinja -np 4 76.9%, no-jinja -np 4 76.9%. The -np hypothesis came from this fleet's own finding that -np 2 vs -np 1 flipped 7 worldview cases, and it was wrong here -- np=4 and np=1 agree to 0.2 points. The weights file has not been modified since nine minutes before the baseline ran. The manifest's backend.model field is --model-id, which is operator-supplied text, and llama.cpp's /props exposes model_path, model_ftype and chat_template, none of which were being read. Spot-checked six other labels whose weights are still on disk, one recorded suite each, re-measured 2026-09-17: btl-4-q4km 91.7% -> 91.7%, gemma4-12b 93.3% -> 93.3%, qwen3.8-9b-distill 91.7% -> 91.7%, ternary-bonsai-27b 93.3% -> 93.3%, edge0-35b 88.3% -> 88.3%, ornith-35b-a3b-q40 worldview 85.0% -> 85.0%. Six for six, exact. Two runtimes (llama.cpp and mlx_lm), GGUF and MLX weights.",
   "impact": "The board reproduces. MiniCPM5-2B is an isolated bad baseline, not evidence of a systemic problem, and the other 147 labels are not called into question by it. What remains true is that nothing in the stamp could have told you that in advance -- the diagnosis took six control runs because the provenance did not identify the weights. server_runtime() now records model_path, model_ftype, model_alias and a chat_template hash, and the PonyExl3 shim reports the same fields so the two runtimes stamp alike.",
   "action": "Re-measure the pre-2026-09-17 MiniCPM5-2B results before quoting them; treat the rest of the board as sound. When a result cannot be reproduced, exhaust the testable hypotheses one at a time and publish the negative ones -- the -np hypothesis was plausible, documented on this fleet, and false, and stopping at plausible would have put a fabricated mechanism on the board."
  },
  {
   "id": "swiftlm-turboquant-works-but-prefill-disqualifies-it",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "SwiftLM's TurboQuant KV claim reproduces on M1 Max; its prefill and missing prompt cache make it the wrong tool anyway",
   "claim": "SwiftLM matches mlx_lm on short-context decode and on capability, and TurboQuant KV genuinely removes its long-context decode collapse exactly as advertised. It still loses on the thing that decides a context-assembly worker: prefill is 2-6x slower than mlx_lm and there is no prompt cache, so every request re-reads the whole evidence set.",
   "evidence": "Qwen3.6-35B-A3B mlx-community 4-bit, one M1 Max 64GB, identical streaming probe measuring inter-token gaps so prefill is excluded from decode. Short context: SwiftLM 48.9 t/s vs mlx_lm 43.6 t/s, phys_footprint 19.0 GB. At 12.3K: SwiftLM vanilla decode 23.2 t/s, with --turbo-kv 48.6 t/s -- back to its short-context rate, flat as claimed. But mlx_lm does not collapse at all (44.6 t/s at 12.3K), so TurboQuant buys parity rather than advantage. Prefill is the gap: SwiftLM 128.7s TTFT at 12.3K (96 t/s) and 450s at 32K (71 t/s), against mlx_lm's 27.9s cold (442 t/s) and 80.1s at 32K (400 t/s). mlx_lm also caches prompts -- a repeat query on the same 12.3K prefix returned in 0.4s where SwiftLM re-prefilled from scratch. At 32K SwiftLM reported MEM_DEMAND 68.4 GB against 68.7 GB of RAM and returned 6 degenerate tokens where mlx_lm returned 118. Capability is a wash: screening 88.3% SwiftLM, 90.0% mlx_lm, 86.7% llama.cpp GGUF, n=60.",
   "impact": "Closes SwiftLM for the distributed context-assembly design. A worker holding 40-100K of evidence pays 7.5 minutes to first token at 32K and pays it again on every query, which is disqualifying regardless of decode rate. The published independent M1 Max figure of ~20 t/s did not reproduce -- measured 46.4 t/s, 2.3x higher -- so that write-up understates it; the vendor's own decode claims, by contrast, hold up under isolation.",
   "action": "Use mlx_lm for long-context MoE work on Apple Silicon. Revisit SwiftLM only if it gains a prompt cache and faster prefill. When comparing runtimes, measure decode from inter-token gaps and never from total request time: the first pass here read 1.02-2.74 t/s at 12.3K and concluded TurboQuant made things worse, when all of that was prefill folded into the rate and TurboQuant in fact doubled decode."
  },
  {
   "id": "swiftlm-throughput-claims-replicate-memory-claim-does-not",
   "date": "2026-09-17",
   "status": "confirmed",
   "title": "SwiftLM's TurboQuant throughput table replicates on four-generation-old hardware; its memory-saving headline does not",
   "claim": "Running SwiftLM's own model, own prompt construction and own metric, the published TurboQuant decode gains reproduce almost exactly in relative terms. The accompanying claim of a 63% memory reduction does not reproduce at all: TurboQuant delivered the full throughput win while using slightly MORE memory, not less.",
   "evidence": "gemma-4-26b-a4b-it-4bit MLX, replicating README table 'Headline Numbers (4-bit MoE)' with their profile_runner method (prompt = 'apple ' x 0.75*ctx, streamed, max_tokens 60, temp 0.6, TPS = (tokens-1)/(total-TTFT)). M1 Max 64GB vs their M5 Pro 64GB. Decode @512: 49.0 vs 77.5 vanilla, 49.8 vs 77.3 TurboQuant. Decode @40K: 32.9 vs 44.3 vanilla, 50.2 vs 70.1 TurboQuant. TurboQuant gain +52.6% measured against +58.2% published, and it holds 102% of its own short-context rate where theirs holds 90%. Absolute ratio 0.63-0.74 of their figures, consistent with a four-generation GPU gap on a workload their own README calls compute-bound. Peak OS RAM @40K: 21.5 GB vanilla and 21.8 GB TurboQuant here, against 48.7 GB and 18.2 GB published.",
   "impact": "The feature works and the vendor's throughput numbers are honest. Two caveats they do not surface: the 63% memory saving is not reproducible here, so TurboQuant should be adopted for speed rather than footprint; and their tables exclude TTFT by construction, which hides prefill entirely. Measured prefill on 30,014 tokens was 129.6 t/s vanilla and 152.6 t/s with TurboQuant -- 197 to 232 seconds, against a runbook that budgets ~40 seconds for a 40K configuration. TurboQuant improving prefill by 18% is itself undocumented.",
   "action": "Quote SwiftLM's throughput claims as sound and its memory claims as unverified. Scale any Apple Silicon benchmark by GPU generation before calling it a discrepancy: the earlier ~20 t/s M1 Max write-up disagreed with this fleet by 2.3x, and the vendor figures agree once hardware is accounted for, so the outlier was the third-party reproduction rather than the vendor."
  },
  {
   "id": "swiftlm-prefill-transient-is-10x-mlx-lm",
   "date": "2026-09-17",
   "status": "revised",
   "title": "SwiftLM allocates ~10x mlx_lm's transient memory per prefill token, which is what created every context ceiling attributed here to models or hardware",
   "claim": "Measured with a fresh server per data point and a single request each, SwiftLM's peak process footprint rises ~29-34 GB over idle while prefilling a 12,000-token prompt. mlx_lm rises 3.0 GB on the same prompt, on two different models. The transient is roughly linear in prompt length at ~2.5 GB per 1,000 tokens, which pushes a 20 GB model past a 64 GB machine at around 15K tokens.",
   "evidence": "Fresh server, one request, peak phys_footprint sampled at 2 Hz. qwen3.6-35B-A3B-4bit at 12,000 tokens: SwiftLM idle 19.0 -> peak 53.0 GB (delta 34.0), mlx_lm idle 19.0 -> peak 22.0 GB (delta 3.0). gemma-4-26b-a4b-4bit at 12,000 tokens: SwiftLM 14.0 -> 43.0 (delta 29.0), mlx_lm 14.0 -> 17.0 (delta 3.0). SwiftLM's own GPU_MEM stays flat at 19-21 GB (qwen) and 15.6-15.8 GB (gemma) across every context depth, so the growth is host-side and not the GPU KV cache -- its own kv_cache_gb estimate is 0.3 GB. mlx_lm continues to 60,000 tokens at 29.0 GB total and dies near 70,000 on '[METAL] Command buffer execution failed: Insufficient Memory'. The mechanism inside SwiftLM is NOT established: the magnitude is ~3x a full-vocabulary fp32 logit vector per position, but qwen has the smaller vocabulary (248,320 vs 262,144) and the larger transient, so vocabulary does not explain it and the prefill loop in their mlx-swift-lm fork was not read.",
   "impact": "Restores this model as a viable long-context worker: 60K of evidence in 29 GB is exactly the context-assembly requirement, and it was declared impossible on the basis of one runtime's memory behaviour. It also explains the earlier TurboQuant confusion on this model -- every comparison above ~12K was run inside a paging regime created by SwiftLM's own overhead, which is why whichever configuration ran second always lost.",
   "action": "Use mlx_lm for long-context work; treat SwiftLM's context ceiling as a property of SwiftLM until its prefill allocation is understood. Measure memory with a FRESH server per data point: both runtimes retain allocator pools across requests, and reusing one server produced internally contradictory per-token figures here (12,233 KB/token at 3K falling to 1,818 KB/token at 30K on the same server) that are artefacts of pool retention, not measurements of anything."
  },
  {
   "id": "check-measured-kv-against-theoretical-before-believing-a-ceiling",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "A per-token memory cost that disagrees with the architecture is a runtime bug, not a hardware limit",
   "claim": "Before accepting any context ceiling, divide the observed memory growth by the token count and compare it with 2 x layers x kv_heads x head_dim x bytes. Agreement within a small factor means the ceiling is real. A factor of ten or more means the serving stack is the problem.",
   "evidence": "Qwen3.6-35B-A3B 4-bit: theoretical KV 80 KB/token, mlx_lm measured ~200 KB/token (2.5x, ordinary overhead), SwiftLM measured ~2,700 KB/token (34x). The 34x was published here as a hardware ceiling of 14K tokens before the check was run; mlx_lm then held 60K on the same box. The same arithmetic already exists in this fleet's seat_size.py, which prices every model's KV from its GGUF header -- it simply was not applied to a new runtime.",
   "impact": "One minute of arithmetic would have prevented a wrong finding, a wrong architectural conclusion about context-assembly workers, and roughly two hours of measurements taken inside a paging regime that made every A/B meaningless.",
   "action": "Add the theoretical-KV check to any new runtime's first evaluation, alongside the coherence gate. Both are cheap preconditions that catch a whole class of confident wrong answers."
  },
  {
   "id": "bonsai2-ternary-qwen38-pays-its-tax-entirely-in-constrained-output",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output",
   "claim": "Ternary-Bonsai-2-27B is Qwen3.8-27B rebuilt in a Hadamard-rotated ternary basis. Measured against the same base model at Q4_0 on one box, it costs 7.3 points on average -- but the average hides the shape entirely. Four of seven paired suites are at parity or better; the whole loss sits in constrained decoding.",
   "evidence": "PQ2_0 GGUF, 6.7 GB against 15 GB for Qwen3.8-27B Q4_0, llama.cpp-prism with Metal kernels. Coherence gate 12/12 at confidence 0.99. Parity or better: synthesis 83.3% vs 83.3%, toolcall 100.0% vs 100.0%, epistemics 100.0% vs 100.0%, worldview 82.5% vs 80.0%. Losses: toolscale -2.5, grounding 66.4% vs 80.0% (-13.6), nullschema 31.9% vs 69.4% (-37.5). Against Bonsai-1 the generational gain is large: worldview 42.5 -> 82.5, synthesis 66.7 -> 83.3, epistemics 85.4 -> 100.0, screening 93.3 -> 98.3. Throughput 7.2 t/s median over 381 cases, against 7.6 for the Q4_0 and 7.8 for Bonsai-1.",
   "impact": "This is the first ternary artefact on the board that is a viable substitute for a model the fleet already runs, rather than a curiosity -- same base weights, 55% less disk, no measurable loss on free-form generation, tool calling or worldview. The -7.3 mean reproduces the 27B ternary figure from the size-crossed-bitwidth 2x2 exactly, which is independent corroboration of quantisation-tax-scales-with-capability from a different model family. It also refines it: the tax is not spread evenly across capability, it concentrates in tasks where output must satisfy a schema.",
   "action": "Use Bonsai-2 where output is free-form and the footprint matters; keep a higher-precision seat for nullschema-style constrained work and grounding. Also note what it does NOT buy: at 7.2 vs 7.6 t/s it is no faster than the 4-bit model it replaces, because ternary unpack cost cancels the halved memory traffic -- ternary buys footprint, not throughput, and that now holds across two Bonsai generations."
  },
  {
   "id": "quantised-kv-is-off-on-m1-because-one-kernel-path-nans",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "The NaN in chad's wide-S kernel is the speculative-verify path, which is why the M1 Max gets 1.88x from the drafter where an M4 Pro gets 3.5x",
   "claim": "On an M1 Max the fused quantized-KV attention never installs, so the cache stays fp16 at 65,536 B/token instead of 34,816. The cause is narrow: the wide-S kernel used for speculative verification returns NaN at short context, while the narrow decode kernel passes every probe. Because the numeric self-check refuses the whole module on any failure, decode loses a fast path that works.",
   "evidence": "chad 2.1.0, mlx 0.32.2, Qwen3.8-27B UD-Q3_K_XL-DFlash2 (64 layers, 16 full attention, 24 q heads, 4 kv heads, head_dim 256, gqa 6). covers(256,6) is True, so 8-bit is the intended default, but kernel_healthy() returns False and chad-bench reports 'kv cache fp16 (65,536 B/tok)'. Running the self-check probes individually: all 9 narrow decode probes pass (max err 6.1e-3 against tolerances of 4e-3 to 1.6e-2). Of the 7 wide-S probes, two return NaN -- (hq 24, hkv 4, n 9, S 3, float16) and (hq 24, hkv 4, n 100, S 3, bfloat16) -- while the same shapes at n=2570 pass at 3.2e-5. Forcing CHAD_KV_BITS=8 halves the cache to 34,816 B/token but routes through stock unfused quantized attention: prefill drops 82 to 56 tok/s and decode 26.3 to 16.4.",
   "impact": "Quantifies a real gap between two machines in this fleet. On M1 Max the choice is fp16 KV or a 38% decode penalty; on M4 Pro, where chad's own published numbers come from, the fused path presumably installs and gives both the 47% cache saving and the speed. It makes the M4 Pro the better host for this stack for a structural reason rather than raw speed. The guard itself is good engineering -- the kernel does not raise, it poisons logits, so refusing it prevents exactly the plausible-garbage failure this fleet has been burned by twice. It also explains a 2x gap between two machines in this fleet. The probes that NaN are the wide-S shapes -- the speculative VERIFY forward, not decode. With QSDPA refused, verification falls back to stock quantized attention, and the drafter's whole advantage is that verifying k tokens costs little more than decoding one. Measured on M1 Max: serial 14.0 -> drafted 26.3 t/s, 1.88x. chad's published M4 Pro figures on the same pack: serial 17.9 -> drafted 62.9 t/s, 3.51x. Same software, same weights; the difference is whether the fused verify kernel installs. That makes 60+ t/s on an M4 Pro a measured number rather than a claim, and makes the M1 Max structurally the wrong host for speculative decoding on this stack.",
   "action": "Do not run chad's quantized KV on M1-class hardware; take fp16 and the larger cache. The upstream fix is to make the self-check per-path rather than all-or-nothing, so a GPU whose decode kernel is healthy keeps quantized KV for decode and falls back only for speculative verify -- worth reporting, with the probe table above as the repro."
  },
  {
   "id": "ptq1-and-pq2-are-the-same-weights-differently-packed",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free",
   "claim": "For Ternary-Bonsai-2-27B the two prism formats are not two quantisations of different quality. They are the same ternary weights at different packing densities, and they generate identical text. PTQ1_0 at 1.75 bpw is 18% smaller than PQ2_0 at 2.13 bpw and costs nothing.",
   "evidence": "Eight suites, 505 paired cases, same server binary, temperature 0: every accuracy matches to the decimal (nullschema 31.9, synthesis 83.3, toolcall 100.0, toolscale 97.5, worldview 82.5, screening 98.3, epistemics 100.0, grounding 66.4) and 505/505 responses are byte-identical. File sizes 5.5 GB vs 6.7 GB. Both were confirmed as the served artefact through /props model_ftype rather than assumed from the filename.",
   "impact": "Removes a choice that looked like a quality/size trade-off. Anywhere Bonsai-2 is deployed, PTQ1_0 is the correct file. It also explains why the bit-width dimension of the earlier size-crossed 2x2 behaved oddly on ternary-trained models: below the trained precision, moving bits does not move quality, because the levels are already fixed at three.",
   "action": "Ship PTQ1_0, not PQ2_0, for any prism ternary-trained checkpoint. Before running a bit-width A/B on a ternary-TRAINED model, diff the generated text first: identical output means the comparison has no variable in it and the cheaper file wins by default."
  },
  {
   "id": "worldviews-conflicting-arm-conflates-three-skills",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "edge0 scores 0/8 on worldview's conflicting arm and 81% on a probe that varies one thing at a time; the arm was measuring three skills at once",
   "claim": "worldview/2's conflicting arm reported edge0-35b as unable to notice that two records disagree. A 48-case probe that separates the variables says otherwise: edge0 detects conflicts in 72% of cases with zero false alarms, and its one real deficit is temporal. The arm's 8 cases each require conflict detection PLUS entity aliasing PLUS inferring a table column header from a fragment, and a failure on the composite cannot say which part failed.",
   "evidence": "The arm's first case supplies 'gemma4-31b | dense QAT | 7.0 t/s | Control.' against 'gemma-4-31B dense: Decode tg128 15.5 t/s.' and asks for decode speed -- the runbook row never states the figure IS a decode speed, and the two records spell the model differently. Isolating each variable across 48 cases (24 conflict, 16 control, 8 added): aliasing is free for both models (edge0 4/4, ornith 4/4). Headerless table fragments defeat both (edge0 0/4, ornith 1/4). Terse adjacent conflicts, conflicts separated by three filler records, categorical conflicts, minority-of-three, and conflicts buried in paragraph prose are all 100% for edge0 except buried at 3/4. Its single genuine weakness is `implicit`: a later-dated record that disagrees without saying 'corrected', 0/4, where ornith scores 4/4. Overall edge0 39/48, ornith 45/48, neither with a false positive on 16 no-conflict controls.",
   "impact": "Reverses a fleet-assignment decision taken on this board an hour earlier. edge0 was ruled out of the claims/contradictions role in the context-assembly design on the strength of 0/8; it is in fact competent at it and should be ruled out only for temporally implicit conflicts. It also questions the arm itself: both models decline to call a headerless table row a conflict, which is defensible reading rather than failure, so the arm may be penalising caution.",
   "action": "Treat any single-arm score of 0 on 8 cases as a hypothesis, not a result -- especially where the cases bundle several skills. Add a false-positive control to any detection suite: half of this probe's cases contain no conflict, and without them a CONFLICT-always model would have scored 56%. The probe is at benchmarks/probes/contradiction.py with its grader self-tested 8/8."
  },
  {
   "id": "temporal-conflict-detection-exists-in-two-seats-not-one",
   "date": "2026-09-18",
   "status": "revised",
   "title": "Two seats detect temporal conflicts, not one: k2-horizon-36b matches ornith at 4/4",
   "claim": "Detecting a conflict where one record simply dates itself later and disagrees is something ornith-35b-a3b and k2-horizon-36b-a4b both do at 4/4. The other three seats score 0-50%. No seat produces false positives, so the fleet's failures are uniformly missed conflicts rather than invented ones.",
   "evidence": "contradiction/1 probe, 48 cases (32 conflict across 8 shapes, 16 no-conflict controls), identical prompt and grader for every seat. implicit arm: ornith 4/4, bonsai-2 PTQ1_0 2/4, btl-4 1/4, edge0-35b 0/4, k2-horizon-3.7b 0/4. Overall conflict detection: ornith 91%, btl-4 78%, bonsai-2 75%, edge0 72%, k2 12%. Controls: 16/16 for all five. The headerless arm -- a figure in a table row whose column header is absent -- is 0/4 for four seats and 1/4 for ornith, a uniformity that indicates the case rather than the models. Added 2026-09-18: k2-horizon-36b-a4b Q4_0, same probe, implicit 4/4, overall 39/48 (81%), controls 16/16, headerless 0/4 like everyone else. The original version of this finding named ornith as the only capable seat because k2-36b had not been run -- it was tested only in its 3.7B form, which scores 12% on conflict detection and is a tool seat.",
   "impact": "Settles a role assignment: chronology/change and claims/contradictions cannot be delegated to edge0, bonsai-2 or k2-horizon as the context-assembly design assumed. The capability is scarce rather than absent, which weakens but does not remove the argument for bi-temporal storage -- the capability the design most needs from its workers is the one the fleet most lacks, and a graph edge carrying valid_at/invalid_at removes the requirement entirely rather than working around it. k2-horizon at 12% is fine; it is a tool caller and should never be handed this work.",
   "action": "Route temporal-conflict work to ornith or k2-horizon-36b. Do not generalise from a model family's small variant to its large one: k2-horizon-3.7b at 12% and k2-horizon-36b at 72% conflict detection are the same family and not interchangeable evidence."
  },
  {
   "id": "edge-extraction-fails-by-asserting-hedged-statements-not-by-malforming-json",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "Every seat emits valid typed edges; three of five turn a hedged future statement into an asserted fact",
   "claim": "The fleet's nullschema scores of 31.9-44.4% suggested it could not produce schema-valid output, and that inference was wrong. Asked unaided to extract typed edges as JSON, seats score 65-95%, with valid JSON and correct keys essentially universal. The real risk in graph ingest is not malformed output but over-extraction: a document saying a figure 'may rise to 60 t/s once the drafter lands' becomes an asserted edge on three of five seats.",
   "evidence": "edgefill/1, 20 cases across 5 arms, same prompt and grader for every seat, no constrained decoding -- which is what allows an MLX seat to be compared with llama.cpp seats at all. Overall: edge0-35b 19/20, ornith 18/20, btl-4 16/20, bonsai-2 PTQ1_0 16/20, k2-horizon-3.7b 13/20. The simple, multi, empty and temporal arms are 100% for four of five seats, including correctly returning [] when nothing is extractable and placing an ISO date in valid_from. Every failure on btl-4 and bonsai-2 is the distractor arm, reason wrong_count_2_want_1: the speculative clause is emitted as a second edge. edge0 and ornith hold it at 3/4. The grader tolerates code fences deliberately, so these are content failures, not formatting ones.",
   "impact": "Removes the main capability objection to a Graphiti-shaped ingest layer -- edge filling is well within reach and edge0, the best extractor on the board, is also the best at it. It relocates the risk: schema validation cannot catch this failure, because a hedged statement rendered as an edge is perfectly well-formed. It enters the store indistinguishable from a measured fact and, unlike a query-time error, is never revisited.",
   "action": "Do not rely on schema validation to protect the graph; it only catches shape. Add an explicit modality gate at ingest -- reject or flag any candidate edge whose source span contains hedging (may, could, expected, planned, future, once) -- and keep the distractor arm as a standing test of that gate. Route extraction to edge0 or ornith; btl-4, bonsai-2 and k2-horizon should not write edges unsupervised."
  },
  {
   "id": "edge0s-small-memory-needs-its-own-runtime-and-the-prerouter-costs-prefill",
   "date": "2026-09-18",
   "status": "confirmed",
   "title": "edge0 is 5.3 GB on its own runtime and 18.4 GB on mlx_lm; the prerouter makes prefill slower, not 59% faster",
   "claim": "The model's small-memory premise is real but belongs entirely to the edge0 framework, which streams expert weights from SSD as they are routed. Served through mlx_lm the same checkpoint is an ordinary 18.4 GB resident model, because mlx_lm reads every expert into Metal buffers at load and streams nothing. Separately, the prerouter that motivated interest in this model does not deliver a prefill gain here -- it costs prefill and returns a smaller decode gain.",
   "evidence": "Same checkpoint, two runtimes, one box. Native `edge0 serve`: phys_footprint 450 MB at load, 5.3 GB after a 128-token generation, decode ~15 t/s, prefill ~378 t/s at 6.5K. Under mlx_lm: 18.4 GB at load before any work, rising to 31 GB under sustained 32K load, decode 53.5 t/s. vmmap shows why -- under mlx_lm the weights sit in IOAccelerator (graphics) regions, 18.2 GB resident with zero open safetensors handles, so nothing is file-backed or evictable. Prerouter A/B on the native runtime, TTFT then decode: 128 tok 2.8s/14.3 vs 2.1s/12.9 off; 1.3K 5.4s/15.6 vs 4.2s/14.4 off; 6.5K 17.2s/14.9 vs 16.4s/14.7 off. The model card's 2.9 GiB figure is qualified 'short contexts; long contexts add KV cache'.",
   "impact": "Restores the four-workers-per-box premise: at 5.3 GB each, four native edge0 seats fit in 21 GB, where four mlx_lm seats would need 74 GB. It also corrects a correction -- the 5.3 GB figure already on this board was right, and today's 18-31 GB measurements were a different runtime rather than a better measurement of the same one. The cost of streaming is 3.5x throughput, 53.5 t/s down to 15, which makes it the wrong choice for an interactive seat and the right one for parallel extraction workers.",
   "action": "Serve edge0 through `edge0 serve`, never mlx_lm, unless the 13 GB is genuinely free and latency matters more than capacity. Re-check any edge0 speed or memory figure for which runtime produced it. Do not plan around the prerouter's advertised prefill gain; leave it on for the small decode benefit or off if TTFT dominates. Note the framework pins mlx-lm 0.31.0, which PyPI has yanked for batched KV cache cross contamination -- single-stream use is unaffected, batching is not."
  },
  {
   "id": "logit-read-decisions-work-on-llama-cpp-seats-with-few-shot",
   "date": "2026-09-19",
   "status": "confirmed",
   "title": "A typed decision costs one forward pass on an existing seat: 300 ms, calibratable distribution, no generation -- but only with few-shot examples and only where the runtime exposes logprobs",
   "claim": "The OpenJev pattern -- ask a typed question, set max_tokens=1, read the probability distribution over the option tokens -- works against the fleet's llama.cpp seats as they are already running. Three conditions decide whether it works at all: the request must disable thinking or the read lands on reasoning logits; the prompt must carry few-shot examples or accuracy is near chance; and the serving runtime must return top_logprobs, which llama.cpp does and mlx_lm's server does not.",
   "evidence": "ornith-35b-a3b production seat, supported/contradicted/not-in-evidence/single-source labelling. With thinking on, the single token went to reasoning_content and the answer-letter mass was zero. Thinking off, zero-shot: 2/5 with soft distributions (top option 0.54-0.89), on a task the same model handles at 100% generatively. Adding four few-shot examples: 4/4 with mass 0.90-1.00 on the correct letter, 257-840 ms per decision, max_tokens=1 throughout. The single-source case -- the fleet-wide 25% abstention wall when models must volunteer the hedge in prose -- reads D:0.90 when the hedge is an option to weigh rather than a sentence to compose. mlx_lm returns no logprobs field at all (already recorded by the coherence gate as 'confidence n/a'), so edge0 over HTTP cannot serve this pattern; in-process MLX forward passes can.",
   "impact": "Turns every llama.cpp seat into a decision engine at prefill cost with no memory overhead, and gives the memory architecture its cheapest primitive: superseded?, contradicted?, hedged?, PII? as ~300 ms calibratable reads instead of generated JSON. The abstention-wall result suggests the fleet's worst measured weakness was partly an artifact of asking for prose. Raw softmax mass is not yet trustworthy confidence -- Jev trains for calibration and these seats do not -- but the gauntlet JSONL archive holds thousands of graded cases with which to fit temperature scaling per seat and task.",
   "action": "Build decision workers as thin clients on existing llama.cpp seats: enable_thinking false, few-shot block per decision type, max_tokens=1, top_logprobs>=8, sum mass per option letter. Never serve decisions through mlx_lm HTTP. Before trusting any confidence number, measure ECE against graded gauntlet cases and calibrate; until then treat the distribution as a ranking, not a probability."
  },
  {
   "id": "needle3-routes-tools-at-100ms-and-cannot-hold-semantic-roles",
   "date": "2026-09-19",
   "status": "confirmed",
   "title": "A 34 MB model routes tool calls in ~110 ms but fires on small talk, misses jargon, and scrambles any extraction with more than two roles",
   "claim": "Cactus Needle 3 is a real single-purpose artefact: direct-phrased tool selection at ~110 ms on CPU for 34 MB. Its usefulness ends precisely where the fleet's needs begin -- it invents tool calls on no-op inputs, misses paraphrase and jargon, and cannot keep subject/predicate/object roles straight in extraction, so it cannot gate, cannot route unsupervised, and cannot fill edges.",
   "evidence": "needle3.cact 34 MB via pip, NEEDLE_TELEMETRY=0, M1 Max CPU. Tool selection, 16 cases (12 tool + 4 no-tool controls): 12/16 at 109 ms median. All 4 misses are the disqualifying kinds: 'bounce the edge0 seat' -> send_message, 'kick ornith over, it's wedged' -> search_docs, and two of four no-tool controls produced invented calls ('thanks, that all looks great' -> restart_seat with empty seat; '2+2?' -> get_seat_status(seat='2')). Extraction: bare JSON Schema silently returns null -- extract() wants the schema shaped as a tool with name/description/parameters; correctly shaped, the edgefill probe scores 2/20 with systematically scrambled roles (predicate='22.4', object='falcon-9b', subject='runbook-falcon-9b'). Embeddings: dim 3072, ~5 ms, 2 of 3 near/far sanity pairs ordered correctly, one inverted ('the fleet is healthy' sits closer to 'quarterly tax filing deadline' than to 'all seats are up') with the whole range compressed into 0.93-0.97.",
   "impact": "Closes three tempting assignments in one afternoon: not the k2-3.7b replacement (k2 is 16/16 on these shapes and refuses nothing it should accept), not the ingest gates (a gate that fires on 'thanks' writes garbage into the graph), not the BGE stand-in (inverted neighbours at 0.93+ similarity is worse than nothing for retrieval). The residual niche -- constrained-vocabulary command routing on hardware that cannot host a real model -- exists but is not a niche this fleet has.",
   "action": "Skip Needle 3 for cortext-fleet purposes; keep ~/venvs/needle and ~/models/needle3.cact (34 MB) for the wearable/microcontroller case if one appears. General rule reinforced twice today: a model that returns null or a zero on first contact is measuring your adapter, not the model -- read the raw output and fix the interface shape before filing the score (bare schema vs tool-shaped schema here; array-vs-object and think-tag stripping for k2-36b yesterday)."
  },
  {
   "id": "system-one-decisions-route-by-family-and-monitorability-beats-accuracy",
   "date": "2026-09-20",
   "status": "confirmed",
   "title": "Typed decisions split across a routing table, not a winner; the only arm you can put a threshold on matters more than the most accurate one",
   "claim": "On a 36-case typed-decision bench (evidence/modality/temporal/alias, class-balanced, one gold rendered as both NLI pair and letter question), no arm wins everything and the operational ranking is set by confidence separation, not accuracy. A 395M CPU NLI model (von-1.0) solves the two families LLMs fail; the most accurate LLM is unmonitorable; the seated tools model systematically abstains from temporal judgment rather than erring.",
   "evidence": "Same 36 cases, four arms. Accuracy: gemma4-12b 89% (evidence/modality/temporal all 100%), ornith 81%, von-1.0 75%, k2-3.7b 58%. von-1.0: temporal 8/8 and alias 7/8 -- the two families every LLM sits at 12-50% on -- at 95 ms on CPU (vendor claims sub-25 ms and 8K context; measured 95 ms fp32 unbatched, config says 2,048). von modality 4/8: coin-flip on the ingest gate. Confidence when right/wrong: ornith 0.92/0.69 (usable separation), von 0.89/0.69, gemma 1.00/0.97 (none -- its 100% cannot be monitored for drift), k2 0.40/0.25 diffuse. k2's temporal 12% was challenged as a possible harness inversion and checked: 0/8 A<->B flips; 7/8 answers are UNKNOWN with half the mass off the letters -- systematic abstention, corroborated independently by its 0/4 on the contradiction probe's implicit arm. Two alias golds are contested (instruct-variant identity), a fifth of that family.",
   "impact": "Defines a new worker class and its seating: temporal+alias decisions on short state go to a 1.6 GB CPU tenant at 95 ms, freeing 35B inference for what needs it; the modality gate stays on ornith logit-reads -- not because gemma is less accurate (it is not) but because ornith is the only arm whose confidence could carry a threshold. Monitorability outranks accuracy for any gate that runs unattended. k2-3.7b is confirmed tools-only for anything dated.",
   "action": "Route: temporal/alias (state under ~2K tokens) -> von-1.0; modality gate -> ornith read; evidence check -> gemma read but never consume its confidence; long-evidence temporal -> ornith. Do NOT calibrate on this bench -- 36 cases ranks, it cannot calibrate; ECE and temperature scaling wait on the gold set, which now blocks three jobs (calibration, gliner size choice, tagger comparison). When a score lands below chance, check for harness/polarity disagreement before filing it -- and when the check exonerates the harness, re-characterise the failure (wrong vs abstaining) because the two have different operational meanings."
  },
  {
   "id": "decision-profiles-follow-training-data-not-architecture",
   "date": "2026-09-20",
   "status": "confirmed",
   "title": "Two near-identical ModernBERT decision models invert each other's family profile, killing the encoder-prior hypothesis and completing a two-encoder CPU decision layer",
   "claim": "The hypothesis that bidirectional encoders are intrinsically suited to temporal/alias reconciliation is falsified in its strong form. Von-1.0 (395M ModernBERT, NLI-trained) and Laya-421M (ModernBERT, typed-decision RL-trained) score opposite profiles on the same immutable 36 cases: what one aces the other flunks. The decision profile is a property of the TRAINING DATA, not the encoder -- and the two checkpoints happen to be complementary, covering three of four families between them on CPU.",
   "evidence": "Same 36 cases, identical grader. von-1.0: temporal 100%, alias 88%, modality 50%, evidence 67%. laya-421m: temporal 12%, alias 100%, modality 100%, evidence 67%, at 22 ms median with some confidence separation (0.66 right / 0.50 wrong -- it ships per-head calibration temperatures). Laya's below-chance temporal was audited per doctrine: not inversion and not abstention -- flat distributions (top option 0.42-0.54) drifting to neutral/entailment regardless of date order; the concept is absent, matching its training data (choice-following RL) versus Von's (WANLI/MultiNLI entailment). Also run: rlcd-modernbert-151m 'Verdict' via its GLiClass architecture -- 47% overall, chance level, confidence 0.97/0.95 blind; eliminated. openjev skipped (87 GB, 0 downloads). Laya's real state window is 512 tokens (cfg max_len), not the advertised sizes; Von's is 2,048.",
   "impact": "The System-One layer becomes a two-encoder CPU pool: temporal -> von-1.0, alias+modality -> laya-421m, both under 100 ms and under 2 GB combined, with evidence checks staying on gemma logit-reads. Laya at 100% modality with 22 ms latency is the first credible challenger to ornith for the ingest gate -- pending the gold set, since 8 cases cannot promote a safety gate. Strategically: stop hunting for the one true decision model; audition checkpoints per family and expect training-data provenance, not architecture, to predict where each lands.",
   "action": "Bench every new decision checkpoint on the immutable 36 before believing any claim; read its training-data description first and predict the profile -- prediction failures are the informative cases. Below-chance triggers the audit, and the audit now has three distinct outcomes to distinguish: inversion (harness bug), abstention (k2), and concept absence (laya's flat drift). The bigger temporal/alias adversarial suite (100-500 cases) is justified only if the von+laya pairing survives the gold set."
  }
 ]
}