Spark-X2.6-4B

• 2026-10-24 — bench X2.6 under the runbook procedure. Alan: "ok, there's a new spark model, we tested the previous one but it's now the same model with different training." Procedure: the lab's benchmarking runbook. One model at a time.
• 2026-10-24 — quant: the vendor's Q4_K_M, not Q4_0 (Claude). The runbook prefers Q4_0 on the M1, but this bench's question is training, and X2.5 was measured on its vendor Q4_K_M. Matching the format removes the quantisation variable. Speed isn't the question for a 4B. The vendor ships no Q4_0.
• Noted: X2.6's Q4_K_M is 3.54 GB against X2.5's 2.60 GB, with identical tensor shapes. The vendor quantised differently this time. The header (variant doc) confirms BF16 embeddings and Q8_0 attn-output, so "same format" doesn't mean identical bits.
• 2026-10-24 — pre-registered (Claude, before any run).
• Server: llama.cpp-b11846/build/bin/llama-server -m Spark-X2.6-4B-Q4_K_M.gguf -np 1 -c 65536 -fa on -ngl 999 --cache-ram 0 --jinja, KV default f16. Beside the full fleet (VACATE=0; 3.5 GB).
• Benches: the coherence gate, then the gauntlet (15 suites, PAR=1, extraction profile), then the context ladder, 8K to the 1M native window, stopping at the budget or swap guard.
• Comparators:
• X2.5 (spark-x25-4b, 2026-09-10) is the training comparison, but it ran a different configuration: -c 32768, q8_0 KV, b10893. Any difference mixes training with configuration, so it's a hypothesis until X2.5 is re-run under this procedure.
• k2-3.7b holds the tool-dispatch seat (toolcall/toolscale 100/100).
• Flint and LFM2.5 single-slot 64K are the only like-for-like procedure comparators.
• Expectations, each falsifiable:
• toolcall rises from 0/60 to at least 12/60. At minimum, the availability arm, where the prompt says a tool exists, should pass. The vendor's agentic-training claim is aimed squarely at this. If it's still 0/60, the training didn't reach tool perception.
• need and unmentioned arms stay low (≤ 3/12 each). Finding Tool perception is mostly all-or-nothing and tracks the checkpoint, not size or vendor claims: detecting an unannounced tool is a separate capability, and the vendor's "agentic" means harnesses that announce their tools.
• worldview > 20% and synthesis > 28.3%, i.e. the "no value" disposition is reduced. Expected movement is uncertain; this is the training question that needs the X2.5 re-run to settle.
• Coherence COHERENT and thinking-off verified on every suite. The template supports enable_thinking: false.
• Peak RSS under 6 GB at 64K, and single-stream decode above 100 t/s (a 4B; LFM2.5 at 2.45B active ran 102.5, contended).
• What would change the plan: a thinking-off precondition failure (then a harness incident, not a score), or a load failure on b11846.
• 2026-10-24 ~12:24Z — complete; the pre-registered expectations, scored (kept beside the plan above).
• toolcall ≥ 12/60: missed. 0/60, 0/12 in every arm including availability. The training didn't reach tool perception.
• need and unmentioned ≤ 3/12: met (0/12 each), but vacuously, since every arm is 0.
• worldview > 20% and synthesis > 28.3%: met. worldview 40.0, synthesis 81.7, with the configuration and quantisation caveat above. (Settled 2026-10-24 by the X2.5 re-run: configuration is ruled out; synthesis is beyond the intervals, worldview isn't.)
• Coherent and thinking-off: met. COHERENT, and thinking-off verified on every suite.
• RSS under 6 GB, decode above 100 t/s: both missed.
• Peak RSS was 7.8 GB in the gauntlet, though the ladder's 64K rung, with no prompt load, read 5.78.
• Decode was 57.8 t/s. My reasoning was wrong: I compared a 4.4B dense model with a 2.45B-active MoE, but decode here is bandwidth-bound on the bytes read per token, and a dense model reads all of them. 57.8 × 3.54 GB ≈ 205 GB/s.
• Plan deviation: the ladder climbed to 256K, not the 1M native window, because eval/ctx_ladder.py tops out at 262,144.
• 2026-10-24 17:51Z — X2.5 re-run under the same procedure. It settles the training-vs-configuration question above; see "Against X2.5, same procedure" under What we learned.

From the gauntlet of record (single slot, 64K, f16 KV, b11846, 15 suites; coherence COHERENT) and the context ladder. Bundles 2026-10-24-4174a46f and -743cfeb3.

Against X2.5, same procedure (re-run 2026-10-24, spark-x2.5-4b-q4km-np1-c64k-b11846, bundle 2026-10-24-0b6bf0e3). The re-run reproduces X2.5's 2026-09-10 run case for case (272/272 outcomes, 270/272 identical answers), so configuration is ruled out. The differences are training (base and post-training both changed) plus the vendor's different quantisation mix.

Like for like (single slot, 64K, b11846; Flint and LFM2.5 from their bundles):

• The retraining didn't reach tool perception under toolcall/1. It is still 0/60, and 0/12 even in the availability and conditional arms, where the prompt states the tool exists or gives a rule. It calls tools in toolscale and tools, so the failure is in deciding to call one from a toolcall-style prompt, not in calling one at all.
• toolcall/2 shows what training did change: X2.6 fetches when asked directly (24/24, against X2.5's 0/24), and X2.5 never makes a parsed call at all.
• It did reduce the "no value" disposition, and that is training, not configuration (the re-run above): synthesis +53, epistemics +23, needle +24, nullschema +15. Worldview +20 is within the intervals.
• The cost: it makes things up more. toolcall/2 honesty fell from 76% to 58%, and tagger_nullable (leaving a field empty when the value is absent) from 88.8 to 71.7. X2.5 says "not there"; X2.6 more often supplies a value.
• The strongest 4B-class judge and tagger benched here. It ties Flint on screening (98.3) and epistemics (95.8), and leads all three models on tagger, needle, enrichment and synthesis, at 3.5 GB of weights.
• Weak on contract (29.2) and worldview (40.0) against Flint.
• Speed: dense, so bandwidth-bound. Decode is 57.8 t/s (contended), slower than the 2.5B-active MoEs (Flint 74.2, LFM2.5 102.5), because every token reads all 3.54 GB of weights. Prefill is 465 t/s. X2.5 (2.60 GB) decodes at 58.4, so the larger file costs nothing measurable here.
• Memory: 3.8 GB at 8K to 12.5 GB at 256K (≈ 34.4 KB/token f16); peak RSS 7.8 GB in the 64K gauntlet.
• Read (Claude, one run each): a strong candidate for screening, tagging and long-context recall at a small footprint, but check its abstention where an empty field matters. Not a tool dispatcher: k2-3.7b (toolcall 100) keeps that job, the seat X2.5 was once tested against.

Alan, 2026-10-24: "ok, there's a new spark model, we tested the previous one but it's now the same model with different training." The question: did X2.6's training fix what X2.5 failed? From X2.5's record (2026-09-10):

• toolcall/1 0/60, every case answered_directly, against toolscale 38/40;
• worldview 20% and synthesis 28.3%, a "no value" disposition.

• Training or quantisation? Configuration is ruled out (X2.5 re-run). The two vendor files still differ in precision (BF16 embeddings and a Q8_0 output head on X2.6), so a Q8_0 of both, or the BF16 weights, would separate training from quantisation. That's Alan's call.
• Why does toolcall stay 0/12 even in the availability arm, when toolscale and tools show it calling tools, and toolcall/2's direct_ask shows it fetching 24/24? Comparing the toolcall prompt shape with toolscale's would answer it.
• What changed in the base checkpoint? The vendor describes post-training only.
• The memory and quality of the full 1M window are unmeasured.
• decide/1 and edgefill are still to run.