LFM2.5-24B-A2B

• 2026-10-23 21:41Z — download the QAD Q4_0 and bench beside Flint. Alan: "yes, follow new guidelines for benchmarking though". The QAD Q4_0 was picked from the vendor's GGUF set because Q4_0 is this box's fast format and the QAD build is trained for it. Pinned to revision 0912d33141f77e10705345a6732bfd83f0252fdc.
• 2026-10-23 — pre-registered (Claude, before any run).
• Benches, server and comparators: the same as Flint (see Flint's family page, 2026-10-23 entry). The full gauntlet through eval/swapseat_gauntlet.sh with -np 4 -c 65536 -ctk q4_0 -ctv q4_0 -fa on -ngl 999 --cache-ram 0 --jinja, extraction profile, on a localhost port beside the full fleet; then decide/1 and edgefill. Comparators: the kolibri-1 and sakura np4 gauntlets; qwen3.6-35b on grounding, nullschema and screening only (its toolcall row is void).
• Expectations, each falsifiable:
• Single-stream decode faster than Flint's at the same context: smaller weights (13.6 vs 26.0 GB) and similar active parameters.
• Resident memory, measured by footprint, under 17 GB at the gauntlet context. Only 10 of 40 layers hold a KV cache.
• toolcall at or above qwen3.6-35b's last valid llama.cpp score. That score doesn't exist (the 2026-09-09 row is void), so this expectation can't be tested without a qwen3.6-35b re-run. It's recorded as a gap, not a prediction.
• On the 3 comparable suites, grounding ≥ 82.4 and screening ≥ 86.7 (qwen3.6-35b, 2026-08-27). nullschema is the risk: an always-thinking model may not take the profile's thinking-off request, and nullschema needs enforced schemas.
• Risk: "always thinks" may trip the harness's thinking-off precondition. A precondition failure is a harness result and is recorded as an incident, not a score.
• 2026-10-23 22:29Z — np4 gauntlet complete; expectations scored against it. 1 pending; 2 met on RSS (13.0 GB), though the footprint method failed; 3 untestable; 4 missed (grounding 71.2, screening 48.3). Thinking-off held. kolibri-1-q4_0-np4-b11533 is not in the published run log under that label, so it wasn't used.
• 2026-10-23 22:37Z — re-run single slot at 64K (Alan's rule). Alan: "hermes won't load a model with less than 64", then "we should only have one slot running on any benchmark, once tested we decide context and slots if it's worth it". New label lfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846: -np 1 -c 65536, default f16 KV, PAR=1, 15 suites, beside the full fleet. The np4 run stays on record as the 16K measurement, not of record.
• 2026-10-23 23:46Z — paused for Flint. Alan: "once that's done kick of flint". Paused after tagger, tagger_nullable, needle and grounding, and 34 of enrichment's 54 cases. Resumed 2026-10-24 02:03Z under the same label, after Flint.
• 2026-10-24 02:21Z — gauntlet of record complete; the pre-registered expectations, re-scored against it (kept beside the plan above).
• Decode faster than Flint: met. 102.5 vs 74.2 t/s median single-stream, both contended. Prefill 763 vs 345 t/s.
• Resident under 17 GB: met on RSS (13.9 GB peak at 64K f16 KV). footprint can't measure mmap'd weights.
• toolcall vs qwen3.6-35b: untestable, as registered. LFM2.5 scored 100.0, but the comparator row is void.
• grounding ≥ 82.4 and screening ≥ 86.7: both missed. grounding 72.8. screening 25.0 is below chance, so a harness audit came first. The audit traced it to the model emitting tool-call syntax where no tools are offered; the harness is sound (Flint 60/60 well-formed on the same suite). nullschema 44.4 against qwen3.6-35b's 61.1. Thinking-off held on every suite.

From the gauntlet of record, lfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 (single slot, 64K, f16 KV, extraction profile, 15 suites; coherence COHERENT), scored from the result files. Like-for-like comparator: Flint (flint-q4_1-np1-c64k-b11846, the same binary, flags and procedure). qwen3.6-35b's 2026-08-27 runs are shown where they exist; its slots aren't on record.

• A reflexive tool-caller. It calls tools in every toolcall arm, including when they're never mentioned (100), the mirror of Flint (55; 1/12 unmentioned).
• It also emits its native tool-call syntax in suites that offer no tools: 43/60 screening, 30/48 epistemics and 18/40 worldview responses.
• That sinks its judgement scores. Screening is 25% (below chance), with 15/19 on the responses that gave a verdict (from the audit).
• Fastest model measured that night: decode 102.5 t/s and prefill 763 t/s single-stream (contended), at 13.9 GB RSS beside the full fleet.
• Behind Flint on 10 of 14 suites, like for like. It leads only on the tool suites and contract.
• Read (Claude, from two runs): a fast dispatcher for tool-heavy loops, where calling a tool is always right. Not a judge, screener, tagger or extractor. Its tool-call reflex would need suppressing (no tools offered and a strict format), or a parser that rejects stray markup, before it could serve a no-tool role.

Found in the 2026-10-23 release scan as an agent/tool-calling MoE with ~2.5B active, the same profile as Flint. Its card claims it beats our coding driver, qwen3.6-35b, on tool benchmarks. It is benched with Flint against the same comparators. Alan's go-ahead: "yes, follow new guidelines for benchmarking though".

• Whether llama.cpp honours the QAD Q4_0's general.sampling.* metadata as the card says. The harness sets its own temperature, which may override it.
• Whether thinking ON, the model's intended mode, changes the judgement suites or the tool-call reflex. The extraction profile turns it off.
• Whether serving with a strict response format (response_format / json_schema) suppresses the stray tool-call markup in no-tool roles.
• decide/1 and edgefill are still to run.