79 findings.

Typed decisions split across a routing table, not a winner; the only arm you can put a threshold on matters more than the most accurate one

On a 36-case typed-decision bench (evidence/modality/temporal/alias, class-balanced, one gold rendered as both NLI pair and letter question), no arm wins everything and the operational ranking is set by confidence separation, not accuracy. A 395M CPU NLI model (von-1.0) solves the two families LLMs fail; the most accurate LLM is unmonitorable; the seated tools model systematically abstains from temporal judgment rather than erring.

Same 36 cases, four arms. Accuracy: gemma4-12b 89% (evidence/modality/temporal all 100%), ornith 81%, von-1.0 75%, k2-3.7b 58%. von-1.0: temporal 8/8 and alias 7/8 -- the two families every LLM sits at 12-50% on -- at 95 ms on CPU (vendor claims sub-25 ms and 8K context; measured 95 ms fp32 unbatched, config says 2,048). von modality 4/8: coin-flip on the ingest gate. Confidence when right/wrong: ornith 0.92/0.69 (usable separation), von 0.89/0.69, gemma 1.00/0.97 (none -- its 100% cannot be monitored for drift), k2 0.40/0.25 diffuse. k2's temporal 12% was challenged as a possible harness inversion and checked: 0/8 A<->B flips; 7/8 answers are UNKNOWN with half the mass off the letters -- systematic abstention, corroborated independently by its 0/4 on the contradiction probe's implicit arm. Two alias golds are contested (instruct-variant identity), a fifth of that family.

Action Route: temporal/alias (state under ~2K tokens) -> von-1.0; modality gate -> ornith read; evidence check -> gemma read but never consume its confidence; long-evidence temporal -> ornith. Do NOT calibrate on this bench -- 36 cases ranks, it cannot calibrate; ECE and temperature scaling wait on the gold set, which now blocks three jobs (calibration, gliner size choice, tagger comparison). When a score lands below chance, check for harness/polarity disagreement before filing it -- and when the check exonerates the harness, re-characterise the failure (wrong vs abstaining) because the two have different operational meanings.

On Gemma 4 12B · k2-3.7b · tagger suite

Two near-identical ModernBERT decision models invert each other's family profile, killing the encoder-prior hypothesis and completing a two-encoder CPU decision layer

The hypothesis that bidirectional encoders are intrinsically suited to temporal/alias reconciliation is falsified in its strong form. Von-1.0 (395M ModernBERT, NLI-trained) and Laya-421M (ModernBERT, typed-decision RL-trained) score opposite profiles on the same immutable 36 cases: what one aces the other flunks. The decision profile is a property of the TRAINING DATA, not the encoder -- and the two checkpoints happen to be complementary, covering three of four families between them on CPU.

Same 36 cases, identical grader. von-1.0: temporal 100%, alias 88%, modality 50%, evidence 67%. laya-421m: temporal 12%, alias 100%, modality 100%, evidence 67%, at 22 ms median with some confidence separation (0.66 right / 0.50 wrong -- it ships per-head calibration temperatures). Laya's below-chance temporal was audited per doctrine: not inversion and not abstention -- flat distributions (top option 0.42-0.54) drifting to neutral/entailment regardless of date order; the concept is absent, matching its training data (choice-following RL) versus Von's (WANLI/MultiNLI entailment). Also run: rlcd-modernbert-151m 'Verdict' via its GLiClass architecture -- 47% overall, chance level, confidence 0.97/0.95 blind; eliminated. openjev skipped (87 GB, 0 downloads). Laya's real state window is 512 tokens (cfg max_len), not the advertised sizes; Von's is 2,048.

Action Bench every new decision checkpoint on the immutable 36 before believing any claim; read its training-data description first and predict the profile -- prediction failures are the informative cases. Below-chance triggers the audit, and the audit now has three distinct outcomes to distinguish: inversion (harness bug), abstention (k2), and concept absence (laya's flat drift). The bigger temporal/alias adversarial suite (100-500 cases) is justified only if the von+laya pairing survives the gold set.

A 34 MB model routes tool calls in ~110 ms but fires on small talk, misses jargon, and scrambles any extraction with more than two roles

Cactus Needle 3 is a real single-purpose artefact: direct-phrased tool selection at ~110 ms on CPU for 34 MB. Its usefulness ends precisely where the fleet's needs begin -- it invents tool calls on no-op inputs, misses paraphrase and jargon, and cannot keep subject/predicate/object roles straight in extraction, so it cannot gate, cannot route unsupervised, and cannot fill edges.

needle3.cact 34 MB via pip, NEEDLE_TELEMETRY=0, M1 Max CPU. Tool selection, 16 cases (12 tool + 4 no-tool controls): 12/16 at 109 ms median. All 4 misses are the disqualifying kinds: 'bounce the edge0 seat' -> send_message, 'kick ornith over, it's wedged' -> search_docs, and two of four no-tool controls produced invented calls ('thanks, that all looks great' -> restart_seat with empty seat; '2+2?' -> get_seat_status(seat='2')). Extraction: bare JSON Schema silently returns null -- extract() wants the schema shaped as a tool with name/description/parameters; correctly shaped, the edgefill probe scores 2/20 with systematically scrambled roles (predicate='22.4', object='falcon-9b', subject='runbook-falcon-9b'). Embeddings: dim 3072, ~5 ms, 2 of 3 near/far sanity pairs ordered correctly, one inverted ('the fleet is healthy' sits closer to 'quarterly tax filing deadline' than to 'all seats are up') with the whole range compressed into 0.93-0.97.

Action Skip Needle 3 for cortext-fleet purposes; keep ~/venvs/needle and ~/models/needle3.cact (34 MB) for the wearable/microcontroller case if one appears. General rule reinforced twice today: a model that returns null or a zero on first contact is measuring your adapter, not the model -- read the raw output and fix the interface shape before filing the score (bare schema vs tool-shaped schema here; array-vs-object and think-tag stripping for k2-36b yesterday).

On k2-3.7b

A typed decision costs one forward pass on an existing seat: 300 ms, calibratable distribution, no generation -- but only with few-shot examples and only where the runtime exposes logprobs

The OpenJev pattern -- ask a typed question, set max_tokens=1, read the probability distribution over the option tokens -- works against the fleet's llama.cpp seats as they are already running. Three conditions decide whether it works at all: the request must disable thinking or the read lands on reasoning logits; the prompt must carry few-shot examples or accuracy is near chance; and the serving runtime must return top_logprobs, which llama.cpp does and mlx_lm's server does not.

ornith-35b-a3b production seat, supported/contradicted/not-in-evidence/single-source labelling. With thinking on, the single token went to reasoning_content and the answer-letter mass was zero. Thinking off, zero-shot: 2/5 with soft distributions (top option 0.54-0.89), on a task the same model handles at 100% generatively. Adding four few-shot examples: 4/4 with mass 0.90-1.00 on the correct letter, 257-840 ms per decision, max_tokens=1 throughout. The single-source case -- the fleet-wide 25% abstention wall when models must volunteer the hedge in prose -- reads D:0.90 when the hedge is an option to weigh rather than a sentence to compose. mlx_lm returns no logprobs field at all (already recorded by the coherence gate as 'confidence n/a'), so edge0 over HTTP cannot serve this pattern; in-process MLX forward passes can.

Action Build decision workers as thin clients on existing llama.cpp seats: enable_thinking false, few-shot block per decision type, max_tokens=1, top_logprobs>=8, sum mass per option letter. Never serve decisions through mlx_lm HTTP. Before trusting any confidence number, measure ECE against graded gauntlet cases and calibrate; until then treat the distribution as a ranking, not a probability.

edge0 scores 0/8 on worldview's conflicting arm and 81% on a probe that varies one thing at a time; the arm was measuring three skills at once

worldview/2's conflicting arm reported edge0-35b as unable to notice that two records disagree. A 48-case probe that separates the variables says otherwise: edge0 detects conflicts in 72% of cases with zero false alarms, and its one real deficit is temporal. The arm's 8 cases each require conflict detection PLUS entity aliasing PLUS inferring a table column header from a fragment, and a failure on the composite cannot say which part failed.

The arm's first case supplies 'gemma4-31b | dense QAT | 7.0 t/s | Control.' against 'gemma-4-31B dense: Decode tg128 15.5 t/s.' and asks for decode speed -- the runbook row never states the figure IS a decode speed, and the two records spell the model differently. Isolating each variable across 48 cases (24 conflict, 16 control, 8 added): aliasing is free for both models (edge0 4/4, ornith 4/4). Headerless table fragments defeat both (edge0 0/4, ornith 1/4). Terse adjacent conflicts, conflicts separated by three filler records, categorical conflicts, minority-of-three, and conflicts buried in paragraph prose are all 100% for edge0 except buried at 3/4. Its single genuine weakness is `implicit`: a later-dated record that disagrees without saying 'corrected', 0/4, where ornith scores 4/4. Overall edge0 39/48, ornith 45/48, neither with a false positive on 16 no-conflict controls.

Action Treat any single-arm score of 0 on 8 cases as a hypothesis, not a result -- especially where the cases bundle several skills. Add a false-positive control to any detection suite: half of this probe's cases contain no conflict, and without them a CONFLICT-always model would have scored 56%. The probe is at benchmarks/probes/contradiction.py with its grader self-tested 8/8.

On edge0-35b · worldview suite

The NaN in chad's wide-S kernel is the speculative-verify path, which is why the M1 Max gets 1.88x from the drafter where an M4 Pro gets 3.5x

On an M1 Max the fused quantized-KV attention never installs, so the cache stays fp16 at 65,536 B/token instead of 34,816. The cause is narrow: the wide-S kernel used for speculative verification returns NaN at short context, while the narrow decode kernel passes every probe. Because the numeric self-check refuses the whole module on any failure, decode loses a fast path that works.

chad 2.1.0, mlx 0.32.2, Qwen3.8-27B UD-Q3_K_XL-DFlash2 (64 layers, 16 full attention, 24 q heads, 4 kv heads, head_dim 256, gqa 6). covers(256,6) is True, so 8-bit is the intended default, but kernel_healthy() returns False and chad-bench reports 'kv cache fp16 (65,536 B/tok)'. Running the self-check probes individually: all 9 narrow decode probes pass (max err 6.1e-3 against tolerances of 4e-3 to 1.6e-2). Of the 7 wide-S probes, two return NaN -- (hq 24, hkv 4, n 9, S 3, float16) and (hq 24, hkv 4, n 100, S 3, bfloat16) -- while the same shapes at n=2570 pass at 3.2e-5. Forcing CHAD_KV_BITS=8 halves the cache to 34,816 B/token but routes through stock unfused quantized attention: prefill drops 82 to 56 tok/s and decode 26.3 to 16.4.

Action Do not run chad's quantized KV on M1-class hardware; take fp16 and the larger cache. The upstream fix is to make the self-check per-path rather than all-or-nothing, so a GPU whose decode kernel is healthy keeps quantized KV for decode and falls back only for speculative verify -- worth reporting, with the probe table above as the repro.

PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free

For Ternary-Bonsai-2-27B the two prism formats are not two quantisations of different quality. They are the same ternary weights at different packing densities, and they generate identical text. PTQ1_0 at 1.75 bpw is 18% smaller than PQ2_0 at 2.13 bpw and costs nothing.

Eight suites, 505 paired cases, same server binary, temperature 0: every accuracy matches to the decimal (nullschema 31.9, synthesis 83.3, toolcall 100.0, toolscale 97.5, worldview 82.5, screening 98.3, epistemics 100.0, grounding 66.4) and 505/505 responses are byte-identical. File sizes 5.5 GB vs 6.7 GB. Both were confirmed as the served artefact through /props model_ftype rather than assumed from the filename.

Action Ship PTQ1_0, not PQ2_0, for any prism ternary-trained checkpoint. Before running a bit-width A/B on a ternary-TRAINED model, diff the generated text first: identical output means the comparison has no variable in it and the cheaper file wins by default.

On epistemics suite · nullschema suite · toolcall suite · toolscale suite · worldview suite

edge0 is 5.3 GB on its own runtime and 18.4 GB on mlx_lm; the prerouter makes prefill slower, not 59% faster

The model's small-memory premise is real but belongs entirely to the edge0 framework, which streams expert weights from SSD as they are routed. Served through mlx_lm the same checkpoint is an ordinary 18.4 GB resident model, because mlx_lm reads every expert into Metal buffers at load and streams nothing. Separately, the prerouter that motivated interest in this model does not deliver a prefill gain here -- it costs prefill and returns a smaller decode gain.

Same checkpoint, two runtimes, one box. Native `edge0 serve`: phys_footprint 450 MB at load, 5.3 GB after a 128-token generation, decode ~15 t/s, prefill ~378 t/s at 6.5K. Under mlx_lm: 18.4 GB at load before any work, rising to 31 GB under sustained 32K load, decode 53.5 t/s. vmmap shows why -- under mlx_lm the weights sit in IOAccelerator (graphics) regions, 18.2 GB resident with zero open safetensors handles, so nothing is file-backed or evictable. Prerouter A/B on the native runtime, TTFT then decode: 128 tok 2.8s/14.3 vs 2.1s/12.9 off; 1.3K 5.4s/15.6 vs 4.2s/14.4 off; 6.5K 17.2s/14.9 vs 16.4s/14.7 off. The model card's 2.9 GiB figure is qualified 'short contexts; long contexts add KV cache'.

Action Serve edge0 through `edge0 serve`, never mlx_lm, unless the 13 GB is genuinely free and latency matters more than capacity. Re-check any edge0 speed or memory figure for which runtime produced it. Do not plan around the prerouter's advertised prefill gain; leave it on for the small decode benefit or off if TTFT dominates. Note the framework pins mlx-lm 0.31.0, which PyPI has yanked for batched KV cache cross contamination -- single-stream use is unaffected, batching is not.

Every seat emits valid typed edges; three of five turn a hedged future statement into an asserted fact

The fleet's nullschema scores of 31.9-44.4% suggested it could not produce schema-valid output, and that inference was wrong. Asked unaided to extract typed edges as JSON, seats score 65-95%, with valid JSON and correct keys essentially universal. The real risk in graph ingest is not malformed output but over-extraction: a document saying a figure 'may rise to 60 t/s once the drafter lands' becomes an asserted edge on three of five seats.

edgefill/1, 20 cases across 5 arms, same prompt and grader for every seat, no constrained decoding -- which is what allows an MLX seat to be compared with llama.cpp seats at all. Overall: edge0-35b 19/20, ornith 18/20, btl-4 16/20, bonsai-2 PTQ1_0 16/20, k2-horizon-3.7b 13/20. The simple, multi, empty and temporal arms are 100% for four of five seats, including correctly returning [] when nothing is extractable and placing an ISO date in valid_from. Every failure on btl-4 and bonsai-2 is the distractor arm, reason wrong_count_2_want_1: the speculative clause is emitted as a second edge. edge0 and ornith hold it at 3/4. The grader tolerates code fences deliberately, so these are content failures, not formatting ones.

Action Do not rely on schema validation to protect the graph; it only catches shape. Add an explicit modality gate at ingest -- reject or flag any candidate edge whose source span contains hedging (may, could, expected, planned, future, once) -- and keep the distractor arm as a standing test of that gate. Route extraction to edge0 or ornith; btl-4, bonsai-2 and k2-horizon should not write edges unsupervised.

On BTL-4 Compact (IQ2_XXS) · edge0-35b · K2-Horizon-3.7B (IFM) · nullschema suite

A per-token memory cost that disagrees with the architecture is a runtime bug, not a hardware limit

Before accepting any context ceiling, divide the observed memory growth by the token count and compare it with 2 x layers x kv_heads x head_dim x bytes. Agreement within a small factor means the ceiling is real. A factor of ten or more means the serving stack is the problem.

Qwen3.6-35B-A3B 4-bit: theoretical KV 80 KB/token, mlx_lm measured ~200 KB/token (2.5x, ordinary overhead), SwiftLM measured ~2,700 KB/token (34x). The 34x was published here as a hardware ceiling of 14K tokens before the check was run; mlx_lm then held 60K on the same box. The same arithmetic already exists in this fleet's seat_size.py, which prices every model's KV from its GGUF header -- it simply was not applied to a new runtime.

Action Add the theoretical-KV check to any new runtime's first evaluation, alongside the coherence gate. Both are cheap preconditions that catch a whole class of confident wrong answers.

A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output

Ternary-Bonsai-2-27B is Qwen3.8-27B rebuilt in a Hadamard-rotated ternary basis. Measured against the same base model at Q4_0 on one box, it costs 7.3 points on average -- but the average hides the shape entirely. Four of seven paired suites are at parity or better; the whole loss sits in constrained decoding.

PQ2_0 GGUF, 6.7 GB against 15 GB for Qwen3.8-27B Q4_0, llama.cpp-prism with Metal kernels. Coherence gate 12/12 at confidence 0.99. Parity or better: synthesis 83.3% vs 83.3%, toolcall 100.0% vs 100.0%, epistemics 100.0% vs 100.0%, worldview 82.5% vs 80.0%. Losses: toolscale -2.5, grounding 66.4% vs 80.0% (-13.6), nullschema 31.9% vs 69.4% (-37.5). Against Bonsai-1 the generational gain is large: worldview 42.5 -> 82.5, synthesis 66.7 -> 83.3, epistemics 85.4 -> 100.0, screening 93.3 -> 98.3. Throughput 7.2 t/s median over 381 cases, against 7.6 for the Q4_0 and 7.8 for Bonsai-1.

Action Use Bonsai-2 where output is free-form and the footprint matters; keep a higher-precision seat for nullschema-style constrained work and grounding. Also note what it does NOT buy: at 7.2 vs 7.6 t/s it is no faster than the 4-bit model it replaces, because ternary unpack cost cancels the halved memory traffic -- ternary buys footprint, not throughput, and that now holds across two Bonsai generations.

On epistemics suite · nullschema suite · toolcall suite · toolscale suite · worldview suite

One baseline was 25 points off and nothing in the stamp could explain it; the other six reproduced exactly

The recorded minicpm5-2b tagger baseline of 51.7% could not be reproduced. Re-measured today it scores 76.9%, with identical dataset_id, grader_id, suite_version, profile, llama.cpp build, slot count and weights file on disk. Every hypothesis was testable and every one was wrong; what remained was that the stamp does not capture enough to tell the two runs apart.

tagger, MiniCPM5-2B Q8_0, build b10893-c32d1dabe throughout. Board baseline 51.7%. Today: --jinja -np 1 76.7%, --jinja -np 4 76.9%, no-jinja -np 4 76.9%. The -np hypothesis came from this fleet's own finding that -np 2 vs -np 1 flipped 7 worldview cases, and it was wrong here -- np=4 and np=1 agree to 0.2 points. The weights file has not been modified since nine minutes before the baseline ran. The manifest's backend.model field is --model-id, which is operator-supplied text, and llama.cpp's /props exposes model_path, model_ftype and chat_template, none of which were being read. Spot-checked six other labels whose weights are still on disk, one recorded suite each, re-measured 2026-09-17: btl-4-q4km 91.7% -> 91.7%, gemma4-12b 93.3% -> 93.3%, qwen3.8-9b-distill 91.7% -> 91.7%, ternary-bonsai-27b 93.3% -> 93.3%, edge0-35b 88.3% -> 88.3%, ornith-35b-a3b-q40 worldview 85.0% -> 85.0%. Six for six, exact. Two runtimes (llama.cpp and mlx_lm), GGUF and MLX weights.

Action Re-measure the pre-2026-09-17 MiniCPM5-2B results before quoting them; treat the rest of the board as sound. When a result cannot be reproduced, exhaust the testable hypotheses one at a time and publish the negative ones -- the -np hypothesis was plausible, documented on this fleet, and false, and stopping at plausible would have put a fabricated mechanism on the board.

On BTL-4 (Q4_K_M) · edge0-35b · Gemma 4 12B · minicpm5-2b · ornith-35b-a3b-q40 · Qwen3.8-9B-Distill (Q8_0) · Ternary-Bonsai-27B (prism-ml) · tagger suite · worldview suite

SwiftLM's TurboQuant KV claim reproduces on M1 Max; its prefill and missing prompt cache make it the wrong tool anyway

SwiftLM matches mlx_lm on short-context decode and on capability, and TurboQuant KV genuinely removes its long-context decode collapse exactly as advertised. It still loses on the thing that decides a context-assembly worker: prefill is 2-6x slower than mlx_lm and there is no prompt cache, so every request re-reads the whole evidence set.

Qwen3.6-35B-A3B mlx-community 4-bit, one M1 Max 64GB, identical streaming probe measuring inter-token gaps so prefill is excluded from decode. Short context: SwiftLM 48.9 t/s vs mlx_lm 43.6 t/s, phys_footprint 19.0 GB. At 12.3K: SwiftLM vanilla decode 23.2 t/s, with --turbo-kv 48.6 t/s -- back to its short-context rate, flat as claimed. But mlx_lm does not collapse at all (44.6 t/s at 12.3K), so TurboQuant buys parity rather than advantage. Prefill is the gap: SwiftLM 128.7s TTFT at 12.3K (96 t/s) and 450s at 32K (71 t/s), against mlx_lm's 27.9s cold (442 t/s) and 80.1s at 32K (400 t/s). mlx_lm also caches prompts -- a repeat query on the same 12.3K prefix returned in 0.4s where SwiftLM re-prefilled from scratch. At 32K SwiftLM reported MEM_DEMAND 68.4 GB against 68.7 GB of RAM and returned 6 degenerate tokens where mlx_lm returned 118. Capability is a wash: screening 88.3% SwiftLM, 90.0% mlx_lm, 86.7% llama.cpp GGUF, n=60.

Action Use mlx_lm for long-context MoE work on Apple Silicon. Revisit SwiftLM only if it gains a prompt cache and faster prefill. When comparing runtimes, measure decode from inter-token gaps and never from total request time: the first pass here read 1.02-2.74 t/s at 12.3K and concluded TurboQuant made things worse, when all of that was prefill folded into the rate and TurboQuant in fact doubled decode.

SwiftLM's TurboQuant throughput table replicates on four-generation-old hardware; its memory-saving headline does not

Running SwiftLM's own model, own prompt construction and own metric, the published TurboQuant decode gains reproduce almost exactly in relative terms. The accompanying claim of a 63% memory reduction does not reproduce at all: TurboQuant delivered the full throughput win while using slightly MORE memory, not less.

gemma-4-26b-a4b-it-4bit MLX, replicating README table 'Headline Numbers (4-bit MoE)' with their profile_runner method (prompt = 'apple ' x 0.75*ctx, streamed, max_tokens 60, temp 0.6, TPS = (tokens-1)/(total-TTFT)). M1 Max 64GB vs their M5 Pro 64GB. Decode @512: 49.0 vs 77.5 vanilla, 49.8 vs 77.3 TurboQuant. Decode @40K: 32.9 vs 44.3 vanilla, 50.2 vs 70.1 TurboQuant. TurboQuant gain +52.6% measured against +58.2% published, and it holds 102% of its own short-context rate where theirs holds 90%. Absolute ratio 0.63-0.74 of their figures, consistent with a four-generation GPU gap on a workload their own README calls compute-bound. Peak OS RAM @40K: 21.5 GB vanilla and 21.8 GB TurboQuant here, against 48.7 GB and 18.2 GB published.

Action Quote SwiftLM's throughput claims as sound and its memory claims as unverified. Scale any Apple Silicon benchmark by GPU generation before calling it a discrepancy: the earlier ~20 t/s M1 Max write-up disagreed with this fleet by 2.3x, and the vendor figures agree once hardware is accounted for, so the outlier was the third-party reproduction rather than the vendor.

PQ2_0 and PTQ1_0 cannot post-quantise a trained model, and selectivity does not rescue them

prism's PQ2_0 (2.13 bpw) and PTQ1_0 (1.75 bpw) work as containers for a model that was TRAINED ternary, and fail as post-training quantisers for one that was not. Restricting them to routed-expert gate/up projections, giving them a 99.6%-coverage imatrix, and backing the bit width off from 1.75 to 2.13 all fail to change that. There is no path to shrinking Ornith below its Q4_0 seat with these formats.

Ornith-35B-A3B (hybrid MoE: 41 layers, 30 SSM + 11 attention, 92.8% of weight in routed experts), converted HF to Q8_0 GGUF, imatrix over 220 chunks of generic corpus, held-out perplexity on 120 chunks of unseen prose. Q8_0 reference 35.2 GB PPL 6.83. All-experts Q4_0 control 20.3 GB PPL 7.15 (+4.7%), coherence 8/8 at confidence 0.99. Ternary on 21 of 41 layers 16.2 GB PPL 19.70 (+188%). PQ2_0 on all gate/up 14.7 GB PPL 43.47 (+536%). PTQ1_0 and PQ2_0 variants both scored exactly 50% on eight closed-set probes -- chance -- and failed the SAME three probes identically despite a 22% bit-width difference, which is systematic damage rather than quantisation noise. The q4_0 control isolates it: same requantise path, same imatrix, same recipe mechanism, same hybrid architecture, and it passes clean.

Action Do not post-quantise to PQ2_0/PTQ1_0. Treat a ternary format as usable only where the weights were trained for it. When testing any quantisation, always run an all-Q4_0 control through the identical pipeline first -- it is what distinguishes a broken format from a broken toolchain, and it took one minute.

On toolcall suite

EXL3/QTIP at 4 bpw costs nothing measurable in capability and three quarters of the throughput

EXL3 at 4.01 bpw on the linear layers scores within noise of the same model at GGUF Q8_0 across eight suites measured the same day on one box, while running at roughly a third of the speed. The quantisation is close to free; the runtime is not.

MiniCPM5-2B, nine suites through the harness against PonyExl3 on Apple Silicon, paired against GGUF Q8_0 re-measured today at -np 1. Mean delta -1.6 points over eight paired suites (epistemics +2.1, screening +10.0, tagger -0.4, tagger_nullable -6.7, enrichment +1.9, grounding +4.0, synthesis -3.3, worldview -20.0). Throughput 19.8-23.0 t/s against 71.2-77.2 t/s for llama.cpp Q8_0 on the same machine, a consistent 3.2-3.7x gap. needle 85.5% at n=110 with prompts to 33.6k tokens is EXL3-only; there is no GGUF pairing for it. shapegen refused to run: its cases have drifted from the dataset lock without a SUITE_VERSION bump.

Action Use EXL3 where footprint is the constraint and latency is not. Do not quote the earlier 27.4 t/s EXL3 figure -- it was measured through a server that never stopped generating.

On epistemics suite · shapegen suite · tagger suite · tagger_nullable suite · worldview suite

A controlled 2x2 confirms it: the 27B loses 7.3 points to 1-bit, the 8B loses nothing

Aggressive quantisation costs capability in proportion to the capability present. Crossing model size with bit width on one family, the larger model degrades and the smaller one does not measurably change -- because it had nothing to lose on the suites where the larger one fell.

prism-ml Bonsai, two sizes x two bit widths, 12 paired suites, one harness, one box. 27B ternary (PQ2_0, 2.13 bpw) to 27B 1-bit (Q1_0, 1.13 bpw): mean -7.3 points. 8B ternary to 8B 1-bit: mean +1.3, inside noise. Per suite the pattern is explicit -- contract 75.0 to 25.0 on the 27B (-50.0) against 22.9 to 20.8 on the 8B (-2.1); synthesis 66.7 to 35.0 (-31.7) against 60.0 to 65.0 (+5.0). toolcall is 100.0 at both bit widths on the 27B and 0.0 at both on the 8B: a capability that was never present cannot be compressed away.

Action When assessing a quantisation, compare per-suite against the SAME model at higher precision, never across models. Expect the largest drops on the suites where the full-precision model scores highest, and treat a flat degradation profile as evidence of a weak baseline rather than a good quantisation.

On toolcall suite

A destroyed model and an incapable one produce the same zero, so the harness now refuses to run without a coherence check

A quantisation that destroys a model does not announce itself in any score. It produces 0% on tool calling, 0% on chainwalk, near-zero everywhere, and those numbers are indistinguishable in a results file from a model that is merely weak at those tasks. The distinguishing evidence is only ever in the generated text, which nobody reads once a run has produced a tidy table. A closed-set precondition separates the two cases in under a second: ask questions whose answer space has two members, then check that the answer is IN the set, that it is the RIGHT member, and that the token distribution is peaked rather than flat.

PTQ1_0 MiniCPM5-2B (1.75 bpw blanket ternary, 772 MB) ran at 105.7 t/s and scored toolcall 0.0%, which was filed and quoted before anyone read its output: 'YHYHYHYHYHRqRqRqRq...' and, on an earlier sample, '不起 priorichter不起vern'. Eight closed-set probes against those same weights: membership 0/8, accuracy 0/8, mean top-token probability 0.08. The same probes against Ornith-35B-A3B Q8_0: membership 8/8, accuracy 8/8, top-token probability 0.94, and bit-identical across three consecutive runs including the confidence to three decimals. Healthy-path cost 0.9-1.3s wall clock including interpreter start. A merely-weak model answering one probe wrong still passes at 7/8.

Action Run the coherence gate before trusting any score from a newly quantised, newly converted or newly downloaded artefact, and treat a zero from an ungated run as unproven. Never report confidence as a sole failure and never report a thinking-budget overrun as bad weights: the first version of this check sent 24 tokens with thinking enabled and scored healthy Ornith-Q8 at membership 0%, because the entire budget went to reasoning_content and content came back empty. A precondition that fails good models gets switched off, and then the thing it guards against happens anyway. --no-coherence exists for base/completion models and is recorded in the manifest. And treat a pass as a floor, never a quality signal: the partial-ternary Ornith passed this gate at 7/8 with confidence 0.64 while carrying 2.9x the perplexity of its Q8 reference. The gate separates rubble from usable; it does not rank the usable.

On chainwalk suite · toolcall suite

An mmapped 19.6 GB checkpoint charges 5.3 GB, plateaus in 250 tokens, and reads as 20 GB if you look at the wrong column

For an SSD-streamed MoE the number that matters is phys_footprint, not PhysMem used and not RSS. Edge0-35B-A3B commits 5.3 GB and stops; the rest is reclaimable page cache that only accumulates because RAM is idle.

Measured with macOS `footprint -p`, which excludes clean file-backed pages. At load: 407 MB. After 250 tokens: 5,269 MB. Across the next 2,250 tokens it moved 8 MB -- 5,343 / 5,349 / 5,342 / 5,346 / 5,347 / 5,355 / 5,351 MB. Flat. Meanwhile file-backed grew 16.9 to 19.9 GB, the whole checkpoint, and RSS read 11.84 GB. PhysMem 'used' climbed 17 GB across three generations.

Action Use `footprint -p <pid>` for any mmap-backed model. RSS and PhysMem both overstate it, and the overstatement scales with how much free RAM the host happens to have.

The faster the SSD, the less an expert prerouter is worth -- which inverts who should want it

Prefetching expert weights pays in proportion to the I/O stall it hides. On fast storage there is little stall, so the gain collapses. The machines that benefit most from streaming architectures are the cheap ones, not the good ones.

Edge0 publish 'up to +59 percent decode; the gain grows with storage latency, model size and routed width K'. Measured on cortext -- 20.2 GB/s NVMe, 1.0M IOPS at 4K, the fastest tier in this programme's own storage table -- the same toggle yields +16.7 percent. The vendor claim and this measurement are consistent rather than contradictory: cortext has the least stall available to hide.

Action Record measured storage throughput alongside every streaming-engine result. Do not port a streaming speedup figure between machines.

A trained expert prerouter buys 17% on decode and nothing on prefill

In an SSD-streamed MoE, predicting which experts the router will pick pays only during decode. Prefill sees no benefit at all, because a long prompt touches nearly every expert anyway -- there is nothing left to predict.

Edge0-35B-A3B (Qwen3.6-35B-A3B base, int4 + recover-LoRA, 256 experts, top_k=4), same checkpoint, same box, prerouter toggled with --no-prerouter. Decode, 300-token generations, warm runs: 14.65 tok/s ON against 12.55 OFF, +16.7 percent. Prefill on a 3,725-token prompt: 282.4 ON against 284.9 OFF, and 261.8 against 265.6 -- a 1 percent deficit, inside noise, with OFF marginally ahead both times.

Action When evaluating a streaming MoE engine, measure decode and prefill separately. A single aggregate throughput figure hides which half the engineering actually affects.

MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites

Extends the 2026-09-11 MoE result to a dense model. Serving the same weights through llama.cpp and through mlx_lm produces the same scores; runtime choice is an operational decision about speed and memory, not a capability one.

Ternary-Bonsai-27B, GGUF PQ2_0 on the prism llama.cpp fork against MLX 2-bit on mlx_lm, 12 paired suites. Eight exact ties including toolcall, tools, toolscale, screening, synthesis, worldview and contract. The two 240-case suites agree to within half a point: tagger 96.2 vs 96.5, tagger_nullable 97.9 vs 97.4. Only epistemics (+4.2) and grounding (+2.4) move, both toward MLX, and both reproduced independently on a second machine. Mean delta +0.55. chainwalk excluded: n=2.

Action Continue recording the server build string per run; two llama.cpp builds were compared as one model earlier in this session and only configuration_id caught it. Do not treat an MLX and a GGUF score as different models.

On chainwalk suite · epistemics suite · tagger suite · tagger_nullable suite · toolcall suite · toolscale suite · worldview suite

IQ2_XXS took 27 points off world knowledge and 17 off tool invocation on identical weights

An aggressive IQ quantisation degrades capability far beyond its throughput cost, and the damage is uneven across suites rather than a uniform decline.

BTL-4, same weights, 14 suites, one harness, temperature 0. IQ2_XXS to Q4_K_M: toolcall 76.7 to 93.3 (+16.7), nullschema 36.1 to 44.4 (+8.3), contract 64.6 to 70.8 (+6.2), synthesis 80.0 to 83.3 (+3.3). Not everything improved: worldview 62.5 to 55.0 (-7.5) and tagger_nullable 96.2 to 94.2 (-2.0). epistemics held at 100.0 and enrichment at ~98.7 in both.

Action Screen models on the best available build, not the smallest. Where only an IQ build is published, record the quantisation next to the score and treat the result as a floor.

On epistemics suite · nullschema suite · tagger_nullable suite · toolcall suite · worldview suite

Sparsity rescues a model from K-quant misalignment and not from IQ codebook lookup

The dense byte-alignment penalty splits on MoE. A K-quant costs 1.42x on a dense model and only 1.10x on a sparse MoE. An IQ codebook costs 1.37x dense and roughly 2.1x on the same MoE. Misalignment is a memory-traffic cost and sparsity cuts traffic; codebook lookup is an arithmetic cost paid per ACTIVE weight, which sparsity does not reduce.

Same weights, two builds, one box. BTL-4 (35B total, 2.1B active, 256 experts, 8 routed) served on llama.cpp b10893: IQ2_XXS 23.6 tok/s decode and 38-71 tok/s prefill; Q4_K_M 48.5 tok/s decode and 82-144 tok/s prefill, three runs each within 0.2 percent. That is 2.06x from the quantisation alone. Cross-checked against ornith-35b-a3b Q4_0 (3.0B active) at 53.4 tok/s: from active parameters alone BTL-4 should reach ~76 tok/s, so IQ2_XXS was costing ~3.2x and Q4_K_M ~1.6x. The K-quant half comes from agents-a1-35b Q4_K_M 49.1 against ornith Q4_0 53.4, both 256-expert MoE.

Action Never serve an MoE from an IQ codebook build if any byte-aligned or K-quant build exists. Q4_K_M is an acceptable MoE seat despite the dense 1.42x. Re-read fleet_audit output with this split in mind.

On Agents-A1 35B

Long-context recall does not degrade with depth; misattribution does the damage, and a string comparison catches all of it

The failure mode in long-context retrieval is not forgetting, it is returning a similar fact about a different subject. That error is mechanically detectable without a model, because the model cites the correct source for the wrong subject. Checking that the cited span mentions the queried subject moves needle/1 from 58.2% to 98.2% and costs no true recall.

needle/1, dataset 5c5199b9eee8a595, ornith-1.5-9b Q4_0, 110 cases across 8k and 32k of real turnpair filler at five depths. present 50/50 (10/10 at each of 10/30/50/70/90 percent), absent 10/10, distractor 4/50 (8.0%). Neither depth nor context length moved any arm. All 46 distractor failures cited a turn id, and cited the RIGHT one - the planted decoy - so no fabrication was involved. Applying a subject-grounding admission check (normalise both sides, require every content word of the subject to appear in the cited span) yields 108/110 (98.2%), recovering all 46. Strictness sweep at 0.25/0.50/0.75/1.00 recovers 2/20/38/46 with the rejection count constant at 2, so there is no precision-recall trade in this data. The 2 are cases where the model returned the correct value citing unrelated filler (turn-0060); needle's grader scores those as retrievals by design, so the filter catches two the grader admits - consistent with epistemics' rule that a right answer with a fabricated citation is not a pass. This is verifiable-memory-architecture section 3 applied to values rather than entities. Not caught: CONFLICT, where two contradictory facts are both correctly cited and both admissible. Tool: benchmarks/tools/admit.py

On ornith-1.5-9b · epistemics suite

Quantised KV degrades which source a model cites, not what it answers - and the raw grader cannot see it

q4_0 KV cache costs nothing measurable in answer accuracy and a real, one-directional amount in citation accuracy. Whether quantised KV is 'free' therefore depends entirely on whether the consuming layer admits on citations. The same measurement supports opposite policies.

needle/1, dataset 5c5199b9eee8a595, ornith-1.5-9b Q4_0, 110 cases paired across f16 and q4_0 KV at 8k and 32k over five depths, identical weights, sequential runs, no contention. Graded two ways. Raw grader: f16 64/110, q4_0 61/110, 11 discordant pairs split 7 to 4, McNemar exact p=0.549 - no detectable difference. With citation-grounded admission (tools/admit.py): f16 108/110, q4_0 99/110, 9 discordant pairs split 9 to 0, p=0.004. Every divergence favours f16. Median latency identical at 24.4s vs 24.5s, so the quantisation buys memory and not speed: 25.7 GB to ~11 GB at 131072 context on this model, 128.0 to 32.0 KB per token.

Action Driver seat stays on f16 KV: it feeds FreshLoops and the graph, where admission is citation-grounded, so citation accuracy is load-bearing. Coding seats use q4_0 KV: nothing admits their output on citations, so the memory is bought in a currency they do not spend. kv-q40-driver.sh now refuses without --accept-citation-cost and prints the trade. Not generalised beyond one model and one suite; architectures with windowed or compressed attention were not tested.

On ornith-1.5-9b

A 23% smaller IQ3_S build decodes 27% slower than Q4_0 on the same weights - the codebook costs more than the bytes it saves

IQ codebook quantisation loses more to lookup cost than it gains from a smaller footprint. Choosing a quant by file size is choosing the wrong variable; byte alignment beats bit count across quant families, not just within the K-quant series.

Qwen3.8-27B, identical weights (866 tensors, 27.32B params), llama-bench c32d1dabe (b10893) on a quiet box with all seats stopped, -p 512 -n 128 -r 3. Q4_0 (file_type=2, 14.63 GiB): prefill 118.48 +/- 0.05 t/s, decode 15.26 +/- 0.01. GSQ-RCO IQ3_S (file_type=26, 3.4375 bpw, 11.28 GiB): prefill 100.87 +/- 0.07, decode 11.21 +/- 0.03. So 23% smaller yields 15% slower prefill and 27% slower decode. Effective decode bandwidth: Q4_0 moves 14.63 GiB at 15.26 t/s = ~223 GB/s (56% of the M1 Max's 400 GB/s); IQ3_S moves 11.28 GiB at 11.21 t/s = ~126 GB/s (32% of peak) - inside the 35-48% band byte-alignment-governs-decode records for IQ formats. Error bars across three reps are 0.01-0.07 t/s, so the gap is not noise.

Action Serve Qwen3.8-27B from the Q4_0 build, already on disk. Do not select a quantisation by file size. Extends byte-alignment-governs-decode from the K-quant series to IQ codebook formats: the rule is cross-family.

On nullschema suite · toolcall suite · toolscale suite · worldview suite

TQ2_0 is fast and lossless to pack, but ternary weights cannot be made post hoc

llama.cpp TQ1_0/TQ2_0 are containers for already-ternary weights, not quantisation algorithms. The packing is exact, and naive post-training ternarisation destroys the model.

Read the kernel: quantize_row_tq2_0_ref sets block scale d = amax then rounds x/amax, so weights already of form {-s,0,+s} round-trip exactly, while normal weights put roughly 87 percent of values at zero. Built the pipeline end to end with stock tools: F16 to BitNet absmean ternarisation (31.4 percent zeros) to TQ2_0, 3.0 GB at 2.85 BPW, 50.5 tok/s, beating Q4_0 45.4. Perplexity went from 8.78 to 51702 on the same corpus. A properly trained ternary model, Bonsai-27B TQ2_0, scores 9.67 on that corpus at 40 percent of peak bandwidth.

Action Use ternary models other people trained. Do not build a ternarisation pipeline unless it becomes a research line in its own right.

Speculative decoding is a net loss on this hardware, in every arm tested

Draft acceptance was healthy and throughput still fell. Eighteen arms across two models, three workloads, four draft depths and two mechanisms produced no win.

Qwen3.8-27B dense IQ3_S with in-file MTP head: baseline 11.5 tok/s, best speculative arm 9.1 (0.79x), monotonically worse with depth. gemma-4-26B-A4B with a 0.53 GB EAGLE-3 head: baseline 68.5, best arm 56.4 (0.83x). Acceptance was 2.0 to 2.9 tokens per target forward throughout, so drafts were good; the verify path costs roughly 3x a plain forward. Measured with llama-cli; llama-server batching untested.

Action Stop evaluating speculative decoding on this box unless llama-server batching is shown to differ.

On a sparse MoE, speculative decoding fails even at 96% draft acceptance

Speculation requires draft_cost < target_cost x (k-1)/k. For an MoE the target cost is the ACTIVE read, not the file size, so sparsity removes the headroom speculation needs. Acceptance is not the binding constraint: a draft can predict almost perfectly and still lose.

k2-36b-a4b reads 3.35 GB/token; the smallest same-vocab draft, k2-3.7B, reads 2.90 GB -- 87% of the target. With --spec-draft-p-min 0.7, acceptance reached 0.960 (97/101) and throughput was still 17.59 tok/s prose / 18.75 json against a 34.76 / 32.47 baseline -- 0.51x. Depth 3 at p-min 0.8: 0.938 acceptance, 17.92 / 18.52. Predicted 0.70x from bytes alone; measured 0.51x, the difference being two-model overhead. On the DENSE qwen3.8-27b the same mechanism works but barely: MTP with p-min 0.7, measured against the same file with speculation off, gives 1.066x on JSON and 0.93x on prose. n-gram speculation (no draft model) gives 1.09x on JSON, nothing on prose. SEPARATE FINDING: --spec-draft-p-min defaults to 0.00, which drafts unconditionally and pays a full verify pass for tokens that were never going to be accepted. Setting a 0.7 floor took qwen3.8-27b MTP acceptance from 0.409 to 1.000 and k2 draft acceptance from 0.562 to 0.960. Any speculation benchmark run at the default is measuring wasted drafts.

On Qwen3.8 27B

llama.cpp's --spec-draft-p-min defaults to 0.00, which guarantees wasted drafts

Speculative decoding drafts unconditionally at the default, paying a full verify pass for tokens that were never going to be accepted. Any speculation benchmark run at the default is measuring waste.

qwen3.8-27b MTP: acceptance 0.409 at the default, 1.000 with --spec-draft-p-min 0.7. k2-36b with a 3.7B draft: 0.562 at the default, 0.960 with a 0.7 floor. Throughput on k2 improved from 14.34 to 17.92 tok/s prose with the floor -- still half the 34.76 baseline, because acceptance was never the binding constraint there.

On Qwen3.8 27B

llama.cpp, mlx_lm and rapid-mlx decode at the same speed; only the format differs

On identical weights all three runtimes are within 6%. The large differences attributed to runtime choice on Apple Silicon are quantisation-format differences in disguise.

Qwen3.5-9B, same weights: GGUF Q4_K_M 21.60 tok/s, GGUF Q4_0 44.11, mlx_lm affine-g64 45.30, rapid-mlx affine-g64 46.64. llama.cpp's DEFAULT format is the outlier at half speed; its good format matches both MLX runtimes exactly. rapid-mlx adds no measurable serving overhead over in-process mlx_lm.

Runtime changes HOW a model phrases a refusal, and nullschema scores that as fabrication

The same weights decline by writing 'not mentioned' under llama.cpp and by writing an empty string under MLX. The grader accepts only the latter, so a formatting difference reads as a 2.5x capability difference.

Qwen3.5-9B absent-case failures: GGUF Q4_0 24 of 24 were the phrase 'not mentioned'; GGUF Q4_K_M 17 of 19; MLX affine-g64 1 of 3; MLX affine-g32 0 of 3. Scores 6.9% / 23.6% / 59.7% / 45.8%. The grader classifies a present, non-empty, not-in-document value as `fabricated`, which is the wrong category for a correct refusal. k2-36b and qwen3.8-27b are unaffected -- zero of their failures are verbal declines -- so their comparisons stand. Analyser at benchmarks/tools/decline_forms.py reads stored responses without touching grader_id.

On Qwen3.8 27B · nullschema suite

Qwen3-Next-80B at Q4_0 runs 46.7 tok/s and is workload-flat

The 80B in Q4_0 is the fastest large model on the board and shows almost no variation across workload type.

45.3 GB file, three workloads on one build (c32d1dabe): json 46.6, prose 46.8, code 46.6 tok/s; prompt ingestion 127 to 157 tok/s. The previously recorded 41.4 tok/s IQ4_XS figure came from a different build and IQ4_XS is no longer on disk, so no speedup is claimed.

Action If a Q4_0 versus IQ4_XS comparison is wanted, re-pull IQ4_XS and measure on the same build.

Prompt framing outweighs a 2.7x bit-width range for abstention

Whether a schema permits an empty answer moves abstention far more than quantisation does. On a diffusion model the permit arm was byte-identical from 2-bit to 8-bit while the forbid arm moved 23 points.

LLaDA-8B-Instruct, 13 frozen cases, five quantisations Q2_K to Q8_0. Permit arm (empty string allowed): 92.3% at every rung, with byte-identical extracted values on all 13 cases across the whole range. Forbid arm (non-empty required): 61.5 / 84.6 / 84.6 / 76.9 / 76.9. Detection never breaks -- the absent arm is 4/4 at every rung in both arms. What degrades is willingness to decline under instruction.

MoE decode reaches a third of dense memory bandwidth, and no runtime fixes it

On Apple Silicon, MoE models achieve 22-36% of peak memory bandwidth during single-stream decode while dense models reach 59%. The gap is GPU occupancy, not bandwidth, and is identical across llama.cpp and MLX.

Bytes-per-token computed from each GGUF tensor table. gemma-4-12b dense 59% of 400 GB/s; gemma-4-26B-A4B (8 of 128) 36%; k2-36b-a4b (8 of 100) 31%; REAP qwen3next (10 of 384) 22%. The same REAP prune: llama.cpp 40.27 tok/s, MLX 40.07 tok/s -- 0.5% apart, both at 22%. Sparser routing is consistently worse, which is the direction the industry is moving.

The 'MLX is 3x faster on MoE' claim benchmarks Ollama, not llama.cpp

Raw llama.cpp and raw mlx_lm decode at the same speed on the same weights. The widely repeated 3x figure uses Ollama as its baseline and measures wrapper overhead.

REAP qwen3next, same prune: llama.cpp Metal Q4_K_M 40.27 tok/s vs MLX 4-bit 40.07 tok/s. qwen3.8-27b: llama.cpp Q4_0 15.18 tok/s vs OptiQ/MLX 4-bit 13.90. The cited source reads '130 tok/s vs 43 tok/s for Ollama with the llama.cpp backend'.

On Qwen3.8 27B

The GGUF format result does not transfer to MLX -- symmetric quantisation is the WORST there

Q4_0 (symmetric) beats Q4_1 (asymmetric) in GGUF. In MLX the ordering reverses: affine (asymmetric) beats mxfp4 (symmetric). The GGUF result is a llama.cpp K-quant implementation detail, not a property of quantisation schemes.

Qwen3.5-9B converted three ways and run on nullschema: MLX affine group-64 43/72, affine group-32 33/72, mxfp4 25/72. All three within 7% on speed (45.30 / 44.86 / 42.40 tok/s), so the differences are not throughput artefacts. In GGUF on k2-36b the same axis ran the other way: Q4_0 +9/-1 against Q4_K_M, Q4_1 +1/-4. Also: more scales did not help -- affine group-32 has twice the scales of group-64 and scored lower.

On nullschema suite

Published EAGLE-3 acceptance rates did not survive either check

Documentation claims 75 to 85 percent draft acceptance for EAGLE heads against 55 to 65 for separate draft models. Neither figure held.

K2-Horizon-36B EAGLE3 head ships its own val_metrics.json showing full_acc 0.494, 0.230, 0.108 across positions and cond_acc flat near 0.47, so 49 percent per step. Measured directly on gemma-4-26B-A4B: 31.2 percent per-token acceptance over 32 cycles. Squish own EAGLEHeadRunner implements should_fallback() dropping to NGRAM below a threshold, which presumes poor acceptance is normal.

Action Treat spec-decoding README numbers as marketing. Measure acceptance before modelling throughput.

Diffusion LLMs are compute-bound on Apple Silicon, so quantising them buys nothing

Masked-diffusion decode cost tracks active parameters, not file size. Quantisation reduces memory traffic, which is not the constraint on this hardware, so it changes throughput by roughly zero.

LLaDA-MoE-7B-A1B at BF16 (13.7 GB) 341.30 ms/step vs Q4_K_M (4.2 GB) 343.29 ms/step -- 3.3x less memory traffic, 0.6% difference. The 4.6 GB dense LLaDA-8B runs 1501 ms/step against the 13.7 GB MoE's 341: the three times larger file is 4.4x faster because it activates 1B params instead of 8B. Diffusion trades compute for bandwidth; Apple Silicon offers ~52 FLOP/byte against a datacenter GPU's ~295.

A purpose-built draft gives 1.13x on structured output and loses on prose

Speculative decoding pays on a dense target when the draft is both tiny and trained for the job. Every method tested helps JSON and hurts prose, because draft runs are short-lived in open text.

z-lab DFlash2, a 5-layer draft matching Qwen3.8-27B's vocab and hidden size, converted to GGUF (llama.cpp's converter has first-class DFlash support via --target-model-dir) and quantised to 1.0 GB -- 7% of the 14.6 GB target. Acceptance 0.972. JSON 15.68 vs 13.87 baseline (1.13x); prose 12.43 vs 14.48 (0.86x). A Q8_0 draft at 1.9 GB scored identically (0.986 acceptance, 15.71 JSON), so the cheaper draft wins on footprint. For comparison: MTP 1.066x on JSON, n-gram 1.09x, a 2.5 GB general-purpose draft 1.00x.

Per-token confidence flags hesitant errors and misses confident ones

A per-position confidence trace isolates the decision token from the JSON scaffolding and catches uncertain answers, but fabrications committed at full confidence are invisible to it. It is a triage signal, not a correctness check.

Instrumented llama.cpp's masked-diffusion decoder to record the confidence each token was committed at. In structured extraction the scaffolding tokens sit at 1.000 and the abstention decision alone is uncertain (0.68). But asked for a VAT number from text containing only a company number, the model returned the company number at confidence 1.000. Separately, 91 of 481 tokens in a baseline generation were committed below 0.5 confidence and were precisely the tokens that made the output ungrammatical.

Byte alignment, not bit-width, determines quantised decode speed

Formats storing an integer number of weights per byte decode far faster than formats whose weights span byte boundaries, independent of how many bits they use.

k2-36b, same weights, five formats: Q4_0 (4.5 BPW, 2 weights/byte) 49.67 tok/s; Q4_1 (5.0 BPW, 2 weights/byte) 48.82; Q5_0 (5.5 BPW, spans bytes) 35.20; Q5_1 (6.0 BPW, spans bytes) 34.48; Q4_K_M (4.5 BPW, super-blocks) 39.85. Q4_1 carries half a bit more than Q4_0 and loses under 2%; Q5_0 carries one more bit than Q4_1 and loses 28%. Clean two-group split with no overlap. Q5_0 and Q5_1 are strictly dominated -- larger AND slower than Q4_K_M.

Benchmark tok/s figures are empty-context artefacts; real decode falls 2.7x with context

llama-bench reports decode at zero context. Under realistic agentic context the same model runs at a third of that, and the published figure is never what a user experiences.

k2-36b Q4_0, llama-bench tg64: 49.67 tok/s. Measured through the server on real requests: 43.8 tok/s decode at empty context, 28.8 at ~4k, 16.0 at ~15k. End-to-end at 15k was 3.26 tok/s on a cold cache because prefill dominates; with a warm prefix the same request returns in 5.5s, so prefix caching rescues multi-turn use but not first contact. Also: llama.cpp's `timings` fields are unreliable -- the same workload reported prefill at 226 tok/s and 21.7 tok/s in different runs. Use wall time.

toolscale's low levels are harder than its high levels

The staircase does not increase in difficulty. Low levels are almost entirely near-misses; high levels dilute them among unrelated filler. Weak models score worse at 4 tools than at 64.

Per level (4/8/16/32/64): lfm2.5-8b-a1b 5,6,7,7,8 of 8; k2-36b 6,7,6,8,8. Both are perfect at 64 and fail at 4. The suite always offers the target plus all its near-misses, then pads with unrelated tools. Consequence for MCP surface design: a large tool surface is an ingestion cost (~65 prompt tokens per schema, linear to 64 tools; ~80 s for 400 tools at 330 tok/s) and not a selection risk. Similar tool names are the risk.

On LFM 2.5 8B-A1B · toolscale suite

Tool perception is binary and tracks the checkpoint, not size or vendor claims

A model either reaches for an unannounced tool or it never does. Nothing between, and it correlates with neither parameter count nor agentic marketing.

toolcall/1, 60 cases: k2-3.7b (2.9 GB) 100%, bonsai-27b 100%, qwen3.8-27b 100%, k2-36b 100%; spark-x2.5-4b 0%, qwen3-8b 0%, lfm2.5-8b-a1b 1.7%. Spark-X2.5-4B is marketed for agentic tool use (tau3-bench 30.4, 'native compatibility with agent harnesses including Hermes') and scores 0/60 here while scoring 95% on toolscale. Its vendor claim concerns selection inside a harness loop; toolcall measures perception when nothing announces the tool.

On k2-3.7b · LFM 2.5 8B-A1B · Qwen3-8B · Qwen3.8 27B · Spark-X2.5-4B (XHToken) · toolcall suite · toolscale suite

Quantisation damage is proportional to the capability present

Losing precision harms a model that can abstain and does nothing to one that cannot. This inverts the usual 'larger models withstand quantisation better' reading on this capability.

nullschema, Q4_K_M to Q8_0, same weights each time: K2-3.7B +4.2, K2-7B -1.4, K2-36B-A4B +16.6 with 12 cases fixed and 0 broken, every case kind improving (absent 87.5 to 100, authority_lure 58.3 to 75.0, lure 16.7 to 37.5). The two small models sit at 36-43% and are flat across a 3x precision range; there is little capability there to damage.

On nullschema suite

Q8_0 and BF16 are behaviourally identical; full precision buys nothing

Above Q8 there is no measurable difference on any suite. Memory is better spent on parameters.

K2-3.7B same weights: Q8_0 and BF16 differ on 0 of 72 nullschema cases, and produce identical scores on all five suites (toolcall 100/100, toolscale 100/100, worldview 50.0/50.0, nullschema 43.1/43.1, synthesis 56.7/56.7). K2-7B agrees (37.5/36.1/37.5 across Q4/Q8/BF16). Memory-matched: K2-7B BF16 at 16.8 GB scores nullschema 37.5% and worldview 30.0%; K2-36B Q4_K_M at 20.8 GB scores 54.2% and 52.5%.

On nullschema suite · toolcall suite · toolscale suite · worldview suite

The 6.7 GB tagger is the most epistemically reliable model in the fleet

gemma4-12b leads both discriminating suites and is the only model that flags single-source evidence, at a fifth of the memory of the models it beats.

worldview 39/40 (97.5%) including single_source 7/8 - every other model scores 0/8 on that arm (see no-model-caveats-single-source-evidence, which was reached before gemma was measured on it). nullschema 63/72 (87.5%) with lure resistance 83.3%. Against k2-36b Q8 at 37.1 GB: 55.0% and 70.8%. Reproduced across two runs and two serving configurations.

On Gemma 4 12B · nullschema suite · tagger suite · worldview suite

gemma4-12b does not act on rules placed in tool descriptions

The same if-then rule scores 12/12 in the system prompt and 0/12 in the tool description on this model, which partially invalidates the docstring strategy for it.

toolcall/1 arms for gemma4-12b: availability 9/12, conditional 12/12, descriptive 0/12, need 12/12, unmentioned 0/12. Every other model tested scores 12/12 on both conditional and descriptive. Rules that must fire on gemma belong in the system prompt. See if-then-rule-drives-tool-invocation, which verified that Pydantic AI passes the full docstring into the tool description - the rule travels, but whether it lands is per-model.

On Gemma 4 12B · toolcall suite

Detecting emptiness and rejecting a plausible candidate are different capabilities

Every model can say 'not present' when nothing is present. Almost none can reject a plausible wrong value that is present. Only the second discriminates.

nullschema by case kind. absent: spark-x2.5-4b 100% at 2.4 GB, gemma4-12b 100%, k2-36b 87.5%. lure: gemma4-12b 83.3%, k2-36b Q8 37.5%, spark 25.0%, k2-36b Q4 16.7%, bonsai-27b 12.5%. Observed in production the same day: K2-36B, given search results mentioning a real company at a similar domain, asserted that intershapes.com links to that company's privacy policy. The page has three external links and none is that.

On Gemma 4 12B · Spark-X2.5-4B (XHToken) · nullschema suite

Speculative decoding does not pay on this box except embedded MTP on llama.cpp

Four separate attempts lose, and the bandwidth explanation from August predicts all of them.

DFlash on qwen3.8-27b: 8.7/8.0/9.0 tok/s against a 12.3/8.9/12.3 baseline across creative, structured and code, retested with parsers correct and reasoning captured separately. DDTree refuses to load ('requires tree_budget'; eligible aliases: qwen3.5-9b-8bit only). MTP never engages through rapid-mlx despite the sidecar shipping in the build. Suffix decoding engages and is flat. The DFlash2 drafter is 3.6 GB against a 15 GB target - 24%, in the range the August ratio rule already flagged as too heavy.

Action Stop testing external-drafter speculation on this hardware unless memory bandwidth changes.

On Qwen3.8 27B

The serving runtime changes tool-calling results more than the model does

Identical weights and a frozen dataset produce wildly different tool-use scores depending only on which server hosts the model.

qwen3-next-80b-reap on toolcall/1 (dataset 10af88c170513155): 47/60 (78.3%) served by rapid-mlx, 21/60 (35.0%) served by oMLX. Per-arm, availability went 12/12 to 0/12 and need 10/12 to 0/12, with 39 cases answering directly instead of calling. Likely mechanism: oMLX applies the model's own chat template, rapid-mlx substitutes its own, and tool rendering is downstream of that.

Action Hold the runtime fixed in any model or quantisation comparison. A model x runtime 2x2 is required before the existing board can be trusted.

On Qwen3-Next-80B-A3B Instruct, REAP pruned · toolcall suite

rapid-mlx drops chat_template_kwargs AND the template's default reasoning block, costing 2x wall clock

Client-supplied template kwargs never reach the template render, so reasoning_effort cannot be set. The template itself IS applied - the earlier claim that it was ignored was wrong.

Same prompt, same weights, both servers. rapid-mlx: 59 prompt_tokens at default, xhigh and low alike - completely flat. mlx_lm: 57 default, 99 xhigh, 87 low, so the template does inject steering text and the kwarg does land. rapid-mlx's 59 sits within two tokens of the UNSTEERED render, not the 99-token xhigh one. Source confirms the intent: vllm_mlx/utils/chat_template.py explicitly passes chat_template_kwargs through, naming reasoning_effort for Qwen3.8, but has an error-driven retry cascade that strips kwargs. Two open upstream issues describe it: #120 (the no-template branch returns plain role:content and drops tools) and #122 (the first retry strips enable_thinking and never restores it). A separate auto-disable heuristic (R12-T2F, 'casual chat request to a thinking-capable model') was investigated and ruled out - it fired once across the substantive batch and the result is identical with or without it. Fair re-measurement 2026-09-09, same curated build on both servers, 3 repeats, identical prompt (the original comparison was confounded: rapid-mlx had the local convert, which has no chat_template.jinja at all, while mlx_lm had the curated build). rapid-mlx 43.6s mean, prompt=49, completion=625, 14.3 t/s. mlx_lm 19.7s mean, prompt=89, completion=279, 14.2 t/s. Decode rate is IDENTICAL - the whole gap is output length. The 40-token prompt difference matches the template's 42-token xhigh steering block, which rapid-mlx does not render. Counterintuitively that block makes output SHORTER ('prioritize correctness, consistency, and clarity'), so the steered model is concise at 279 tokens and the unsteered one rambles to 625.

Action reasoning_effort is plain text - the template injects a single sentence, and medium is the empty string. It can be sent as a system message to any server. Measured recovery on rapid-mlx: unsteered 625 tokens, injected xhigh 500, against mlx_lm's native 279 - so injection recovers about a third, and a system message is not equivalent to the template rendering the same text in its own position. Use mlx_lm where the steering matters.

oMLX degrades models it has no native kernel for, by up to 58 points

oMLX ships architecture-specific kernels for a handful of families and falls back to a generic implementation for everything else. The fallback silently loses accuracy.

granite-guardian-4.1-8b on screening, 60 cases. Quantisation ladder on mlx_lm: GGUF Q4_K_M via llama.cpp 60/60 (100.0%), MLX bf16 unquantised 59/60 (98.3%), MLX 8-bit 59/60 (98.3%), MLX mixed_3_6 57/60 (95.0%). The SAME mixed_3_6 build via oMLX: 22/60 (36.7%). So conversion and 8-bit cost nothing measurable, mixed_3_6 costs about 3 points, and oMLX costs 58. Ruled out: sampling (byte-identical output at temperature 0 seed 42), KV cache (--no-cache reproduces 36.7% exactly), prompt construction (prompt_tokens=71 on both servers for the same request, yet completions differ), oQ (never invoked on the load path), Bonsai shims (gated to 1-2 bit). The pattern that fits: oMLX ships eleven qwen35_* kernels plus qwen38, gemma4, glm_moe_dsa, deepseek_v4, minimax_m3, laguna, llama4 and others, and NOTHING for granite or qwen3_next. qwen3.8-27b (model_type qwen3_5) scored 60/60 on toolcall/1 under oMLX and 60/60 under rapid-mlx - no degradation. The two models that collapsed are the two with no native kernel path. Third reference added 2026-09-09: the same weights run through BARE mlx_lm in-process, no HTTP and no serving layer, score 55/60 (91.7%, [82-96]) - overlapping the mlx_lm server's 95.0% [86-98] within noise at n=60. oMLX's 36.7% [26-49] overlaps neither, nor the GGUF baseline. Upstream MLX therefore computes these weights correctly and oMLX is the outlier against three independent references.

Action Check omlx/patches for the model's architecture before serving it there. granite and qwen3_next have none. Serve those on mlx_lm or rapid-mlx. Note omlx ships mlx_vlm_qwen4_exp_compat, so Qwen3.8-Flash-Next (qwen4exp) may have an MLX path after all, which mlx_lm does not offer. Reproducible case for upstream: granite-guardian mixed_3_6, screening, 95.0% mlx_lm vs 36.7% oMLX.

On granite-guardian · Granite Guardian 4.1 8B · Qwen3.8 27B · toolcall suite

No model qualifies single-source evidence, even when explicitly told to

Models act on the content of evidence, not its provenance tier. Asked to flag a claim supported by only one external document, every model tested stated it as established fact instead.

synthesis/2 (dataset 267d1954e225c5b6), single_source arm, 10 cases each. REAP 60b 0/10, gemma4-12b 0/10, K2-Horizon-7B 0/10, K2-Horizon-3.7B 0/10 - 40 of 40 cases, every failure classified 'overclaimed'. The system prompt instructed it directly: 'If the only support is a single external document, give the answer but say SINGLE-SOURCE and cite it. Do not refuse, and do not state it as established.' The same models score 10/10 on the conflicting and insufficient arms, so they follow CONFLICT and INSUFFICIENT instructions in the same prompt - it is specifically the provenance-tier condition they do not evaluate.

Action Caveat at ingest, deterministically, from the provenance tier the store already holds - not by asking the model to notice. Consistent with okf:verifiable-memory-architecture's conclusion on entity grounding: do not try to make the model stop, reject (or qualify) at the boundary.

On Gemma 4 12B · toolcall suite

Models call tools when given an if-then rule, not when told a tool exists

Announcing availability is close to useless; naming the trigger, the action and the displaced default works.

toolcall/1, 12 cases per arm. Stating a tool is available: qwen3.6-35b 2/12. Stating 'If a value is missing from the document, you SHOULD call the available tool rather than reporting it absent': 12/12 on the same model. Reproduced through Pydantic AI, which passes the full docstring into the tool description: terse docstring 0/6 on both qwen3.6-35b and REAP, if-then docstring 4/6 on both. qwen3.8-27b is the exception, scoring 12/12 on every arm under two different runtimes.

Action Write every tool docstring as an if-then rule. Verify per harness that the full description reaches the model, since some send only the summary line.

On Qwen3.6 35B-A3B · Qwen3.8 27B · toolcall suite

Converting the judge to MLX destroyed it, from 100% to 36.7%

granite-guardian-4.1-8b quantised to MLX mixed_3_6 and served by oMLX loses most of its judging accuracy.

screening, 60 cases, 100% coverage. GGUF Q4_K_M via llama.cpp: 60/60 = 100.0%, reproducing the 2026-08-31 baseline exactly. MLX mixed_3_6 (3.8 GB) via oMLX: 22/60 = 36.7%. Not yet separable into quantisation damage versus runtime, because REAP moved 78.3 to 35.0 on a runtime change alone the same day.

Action Judge stays on the GGUF via llama.cpp. Validate any converted model against its own baseline before it serves. Isolate quant from runtime by serving the same MLX build on mlx_lm.

On Granite Guardian 4.1 8B

Judge and tool-use scores are runtime-sensitive because the decisions sit on ~0.4 logprob margins

Any task whose output is a decision rather than a passage is decided by a first-token margin narrow enough for kernel-level numerical differences to flip it.

granite-guardian-4.1-8b, identical request to oMLX and mlx_lm: both produce prompt_tokens=71 and the same first token, so template, bos/eos and prompt construction are identical. mlx_lm's top_logprobs on that token: 'Reject' -0.7500, 'Keep' -1.1250 - a margin of 0.375, about 69% against 32%. The verdict is decided entirely by that one token. oMLX does not return logprobs, so the distributions could not be compared directly, only the argmax.

Action Treat runtime as a controlled variable for any decision-shaped suite, the way dataset and grader already are. Record it in the result identity. Prefer a judge on the runtime it was baselined on.

On Granite Guardian 4.1 8B · toolcall suite

Running 32 requests at once is worth 3-4x, and single-stream tok/s is the wrong metric

Aggregate throughput, not decode rate, governs wall-clock on suites and agent fleets.

qwen3.8-27b: 15.3 tok/s at n=1, 62.1 at n=32. REAP: 44.7 to 139.2. Both knee at n=32 and fall off at n=48 (47.5 and 128.9). Wall time is flat from n=8 to n=32, so batching above n=8 is nearly free; the cost is all paid between n=1 and n=8.

Action Run bulk extraction at n=32. Note oMLX does not batch (16.3 aggregate at n=32 with the cap raised), so bulk work belongs on mlx_lm or rapid-mlx.

On Qwen3.8 27B

Three different bottlenecks, depending on how the model is served

Dense models are bound by memory bandwidth, mixture of experts models by active parameters, and streamed models by SSD throughput times cache hit rate. The same machine gives very different ceilings depending on which applies.

Measured on one M1 Max with 64 GB: memory bandwidth about 400 GB/s, sequential SSD read 6.34 GiB/s, a ratio of roughly 63 to 1. Dense 30B models run 9 to 11 tok/s. MoE models with 3B active run 37 to 52 tok/s at similar total size. A published SSD-streamed build measured 5.33 tok/s on a 12.47 GiB/s disk, which scales to about 2.7 tok/s here.

Action Judge a candidate by which bottleneck applies to it, not by parameter count. Slow is acceptable for unattended batch work; exceeding the memory ceiling is not.

The extraction suites can no longer separate the leading models

At n=72, nullschema cannot distinguish 97% from 89%.

qwen3-next-80b 97.2% [90.4,99.2]; granite-4.2-30b 90.3% [81.3,95.2]; gemma4-12b 88.9% [79.6,94.3]. Intervals overlap heavily.

Action nullschema needs more cases or harder tiers before it can adjudicate the top of the field.

On Gemma 4 12B · granite-4.2-30b · Qwen3-Next-80B-A3B Instruct · nullschema suite

Expert pruning cost this model nothing measurable

REAP removes a quarter of Qwen3-Next-80B's parameters and the extraction behaviour is preserved, including the abstention that won it the seat.

Same suites, same grader, same profile, schema enforcement verified. nullschema 70/72 for both, identical failures. grounding 111/125 full against 109/125 pruned. screening 55/60 against 59/60. Throughput 37.7 against 41.4 tok/s. Weights 40 GB against 30.4 GB, 80B params against 60B.

Action Prune before buying memory. This was a 30 GB download that returned more usable capacity than any deletion made today.

On nullschema suite

mlx_lm silently ignores response_format

mlx_lm's OpenAI server accepts response_format and never applies it. Constrained arms become free generation.

response_format and json_schema appear 0 times in mlx_lm/server.py (1904 lines). Granite via MLX scored forbid 24/24, permit 24/24, optional 24/24 - perfectly identical arms; via llama.cpp 22/24, 21/24, 22/24.

Action Guard added to runner.py: suites sending response_format now refuse to run against a server that fails a sentinel-enum probe.

On Qwen3-Next-80B-A3B Instruct · Qwen3.8 27B · nullschema suite

KV cache decides what fits, and only measurement predicts KV

Parameter count does not predict memory at context. Neither does the standard KV formula for every architecture.

Measured on a context ladder from 8k to 131k, as resident memory: muse-glimmer 1.77 GB of KV, glm-4.7-flash 6.67 GB, granite 30.26 GB. Predicting from 2 x layers x kv_heads x head_dim matched granite (0.250 predicted, 0.236 measured) but overestimated muse-glimmer by 6x and glm-4.7-flash by 7x.

Action An earlier version of this finding claimed GLM-4.7-Flash needed 88 GB and could not run here. That was calculated, not measured, and it was wrong by 7x. Head counts in a config do not describe the KV cache for architectures using windowed or compressed attention. Run the ladder.

HTTP 200 does not mean a model is loaded

Readiness probes that check an endpoint rather than generation produce silent garbage runs.

mlx_lm returned 200 on /v1/models while the laguna architecture was unsupported and every request failed; the run hung for 2 hours. Earlier, curl && break accepted a 503 and 317 cases ran against a loading server.

Action Readiness now requires a successful completion, with >=512 max_tokens so reasoning models are not starved.

Schema enforcement is per-model, not per-runtime

llama.cpp applies grammar for some models and silently drops it for others.

Identical strict enum schema: granite-4.2-30b returns {"v":"__sentinel__"}; qwen3-coder-30b enforced; muse-glimmer-30b returns plain text 'blue'.

Action Probe every model, not every runtime. Recorded as a stated gap rather than a number.

On granite-4.2-30b · Muse-Glimmer 30B · Qwen3 Coder 30B · nullschema suite

Speculative decoding did not help, by either method tested

Neither a published DFlash draft nor a matched MTP head improved decode speed on this machine. Both made it slower.

DFlash on muse-glimmer-30b, target and draft from the same publisher: baseline 5.75 tok/s, speculative 2.65, a 2.2x slowdown, with 11.1% draft acceptance over 1654 generated tokens. MTP on qwen3.6-27b under rapid-mlx, one flag apart: 15.35 tok/s without, 13.5 with, 12% slower.

Action An earlier version of this finding blamed a repository mismatch, because the first DFlash test paired an unsloth target with a meta-models draft. Repeating it with both from one publisher gave 11.1 percent acceptance against 10.8. The pairing was never the cause. The draft does not predict this model.

On Muse-Glimmer 30B

Active parameters, not total, set throughput

Dense and MoE models of the same total size differ 3-5x in decode speed.

Dense 30B: granite 9.0, muse-glimmer 9.6 tok/s. MoE 33-35B: ornith 28.5, qwen3.6-35b 52.4 tok/s. qwen3-next-80b (3B active) 37.7 tok/s.

Action Judge candidates on active params for speed and total params for memory.

On Qwen3-Next-80B-A3B Instruct · Qwen3.6 35B-A3B

Specialist labels failed to predict outcomes three times in four

A model tuned for a task does not reliably win that task's suite.

gemma4-12b (general) beats qwen3-next-80b (larger) at tagging by 7.5. Coding models score mid-pack on extraction suites that never measure code.

Action Built shapegen and sitebuild to measure code against a contract instead.

On Gemma 4 12B · Qwen3-Next-80B-A3B Instruct · shapegen suite · sitebuild suite

The tagger fine-tune scored below its own base model

tagger-e2b-v1 (LoRA, 270 turns) is worse than gemma4-e2b, its base.

Same runtime, same profile, same 240 gold pairs, 100% coverage: tagger 85.8% [81-90] base vs 74.2% [68-79] distill. Non-overlapping.

Action Revisit the training data, not the runtime.

On Gemma 4 12B · Gemma 4 E2B · tagger-e2b-v1 (LoRA fine-tune) · tagger suite

Reasoning increases misattribution under authority pressure

Enabling reasoning degrades extraction while leaving detection untouched.

authority_lure 100% to 40%. Screening 91.7% both ways with identical failure sets. qwen3-next-80b-thinking nullschema 58.3% vs 97.2% for Instruct.

Action Reasoning is a dial for agentic work, not a default.

On Qwen3-Next-80B-A3B Thinking · nullschema suite