LFM2.5-24B-A2B · file← LFM2.5-24B-A2B
LFM2.5-24B-A2B
The only file: Liquid AI's own Q4_0, trained for that format rather than rounded down to it. Run twice: four conversations at 16K for reference, then one at 64K, the run of record.
- Size
- 13.6 GB
- Status
- Benched 24 Oct 2026
- Scored runs
- 30
- Smallest Mac
- 24 GB measured, 64K context
| Suite | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| chainwalklfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 50.0% | 9.5–90.5 | Passed 1/2 | llama.cppextraction | 2026-10-24 |
| chainwalklfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 50.0% | 9.5–90.5 | Passed 1/2 | llama.cppextraction | 2026-10-23 |
| contractlfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 75.0% | 61.2–85.1 | Passed 36/48 | llama.cppextraction | 2026-10-24 |
| contractlfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 72.9% | 59.0–83.4 | Passed 35/48 | llama.cppextraction | 2026-10-23 |
| enrichmentlfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 51.9% | 38.9–64.6 | Passed 28/54 | llama.cppextraction | 2026-10-24 |
| enrichmentlfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 38.9% | 27.0–52.2 | Passed 21/54 | llama.cppextraction | 2026-10-23 |
| epistemicslfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 70.8% | 56.8–81.8 | Passed 34/48 | llama.cppextraction | 2026-10-24 |
| epistemicslfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 75.0% | 61.2–85.1 | Passed 36/48 | llama.cppextraction | 2026-10-23 |
| groundinglfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 72.8% | 64.4–79.8 | Passed 91/125 | llama.cppextraction | 2026-10-24 |
| groundinglfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 71.2% | 62.7–78.4 | Passed 89/125 | llama.cppextraction | 2026-10-23 |
| needlelfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 54.5% | 45.2–63.5 | Passed 60/110 | llama.cppextraction | 2026-10-24 |
| needlelfm2.5-24b-a2b-qad-q4_0-np4-b1184650% coverage | 56.4% | 43.3–68.6 | Passed 31/55 | llama.cppextraction | 2026-10-23 |
| nullschemalfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 44.4% | 33.5–55.9 | Passed 32/72 | llama.cppextraction | 2026-10-24 |
| nullschemalfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 38.9% | 28.5–50.4 | Passed 28/72 | llama.cppextraction | 2026-10-23 |
| screeninglfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 25.0% | 15.8–37.2 | Passed 15/60 | llama.cppextraction | 2026-10-24 |
| screeninglfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 48.3% | 36.2–60.7 | Passed 29/60 | llama.cppextraction | 2026-10-23 |
| synthesislfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 75.0% | 62.8–84.2 | Passed 45/60 | llama.cppextraction | 2026-10-24 |
| synthesislfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 70.0% | 57.5–80.1 | Passed 42/60 | llama.cppextraction | 2026-10-23 |
| taggerlfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 62.1% | 55.8–68.0 | Passed 149/240 | llama.cppextraction | 2026-10-23 |
| taggerlfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 62.5% | 56.2–68.4 | Passed 150/240 | llama.cppextraction | 2026-10-23 |
| tagger_nullablelfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 50.0% | 43.7–56.3 | Passed 120/240 | llama.cppextraction | 2026-10-23 |
| tagger_nullablelfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 45.4% | 39.2–51.7 | Passed 109/240 | llama.cppextraction | 2026-10-23 |
| toolcalllfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 100.0% | 94.0–100.0 | Passed 60/60 | llama.cppextraction | 2026-10-24 |
| toolcalllfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 100.0% | 94.0–100.0 | Passed 60/60 | llama.cppextraction | 2026-10-23 |
| toolslfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 86.7% | 62.1–96.3 | Passed 13/15 | llama.cppextraction | 2026-10-24 |
| toolslfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 86.7% | 62.1–96.3 | Passed 13/15 | llama.cppextraction | 2026-10-23 |
| toolscalelfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 100.0% | 91.2–100.0 | Passed 40/40 | llama.cppextraction | 2026-10-24 |
| toolscalelfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 100.0% | 91.2–100.0 | Passed 40/40 | llama.cppextraction | 2026-10-23 |
| worldviewlfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 55.0% | 39.8–69.3 | Passed 22/40 | llama.cppextraction | 2026-10-24 |
| worldviewlfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 27.5% | 16.1–42.8 | Passed 11/40 | llama.cppextraction | 2026-10-23 |
Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.
lfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846
Of record
The measurement of record for this model.
The run’s own note: Single slot at 64K (benchmark rule). Paused after 4 suites and resumed under the same label; scored from result files. screening below chance: the model emits tool-call syntax where no tools are offered (audited; harness sound).
- Server
- 1 slot, 64K of context per request, KV cache f16 (default), flash attention on, 1 suite at a time
- Runtime
- llama.cpp b11846 (1a24df163), plain upstream
- Speed
- decode 102.5 tokens/s, prefill 763.3 tokens/s, medians. Single stream, full fleet resident (contended).
- Peak memory
- 13.9 GB resident, with 64K of context allocated
- Weights
- 13.6 GB file · sha256
d1c03f73ecaf6101… - Coherence check
- passed
- Left out by plan
- shapegen, sitebuild
- Machine
- Mac Studio (M1 Max, 64 GB)
- Released
- 24 October 2026, bundle
2026-10-24-12403c71
Memory by context
The same file and server, started at each context size in turn: what it held in memory and how fast it wrote on a short prompt. It measures memory and serving, not how well the model uses a long context. Released 24 October 2026, bundle 2026-10-24-e06e545c.
The ladder’s own note: Context ladder 8K to native 128K, single slot, f16 KV, beside the full fleet. Decode on a trivial prompt per rung; the 16K rung's decode is a single-sample contention dip.
| Context | Resident memory | Swap in use, whole Mac | Decode | Load |
|---|---|---|---|---|
| 8K | 12.93 GB | 13.57 GB | 109.5 tokens/s | 3 s |
| 16K | 13.05 GB | 13.57 GB | 23.2 tokens/s | 3 s |
| 32K | 13.42 GB | 13.57 GB | 110.6 tokens/s | 3 s |
| 64K | 14.05 GB | 13.57 GB | 109.9 tokens/s | 3 s |
| 128K | 15.3 GB | 13.57 GB | 108.6 tokens/s | 3 s |
lfm2.5-24b-a2b-qad-q4_0-np4-b11846
Reference
Kept for reference. It isn't the measurement of record.
The run’s own note: Not the measurement of record: 4 slots give 16K per request, below the 64K Hermes needs. shapegen and sitebuild failed in the harness (dataset lock / dataset field), unscored. needle: 55 of 110 cases rejected (32K cases exceed the 16K slot).
- Server
- 4 slots, 16K of context per request, KV cache q4_0, flash attention on, 4 suites at a time
- Runtime
- llama.cpp b11846 (1a24df163), plain upstream
- Speed
- decode 32.6 tokens/s per request, prefill 583.8 tokens/s, medians. 4 requests in flight, full fleet resident (contended).
- Peak memory
- 13 GB resident, with 64K of context allocated
- Weights
- 13.6 GB file · sha256
d1c03f73ecaf6101… - Coherence check
- not recorded
- Left out by plan
- none
- Machine
- Mac Studio (M1 Max, 64 GB)
- Released
- 24 October 2026, bundle
2026-10-24-977b4a5a
Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.
| File | LFM2.5-24B-A2B-QAD-Q4_0.gguf |
| Format | GGUF Q4_0 (QAD checkpoint) |
| Where from | downloaded |
| Status | benched 2026-10-24 |
| Hugging Face | LiquidAI/LFM2.5-24B-A2B-GGUF |
How it was made
Downloaded, not made here.
| field | value |
|---|---|
| repo | LiquidAI/LFM2.5-24B-A2B-GGUF |
| file | LFM2.5-24B-A2B-QAD-Q4_0.gguf |
| revision | 0912d33141f77e10705345a6732bfd83f0252fdc |
| size | 13,578,190,880 bytes (ls -l; matches the HF listing) |
| sha256 | d1c03f73ecaf61017d70419eec9e5dc0b05d50ae26c23d522829b934fbabf8a5 (shasum -a 256; equals the HF LFS oid) |
| downloaded | 2026-10-23 21:41–21:50Z, huggingface_hub 1.33.0 with HF_XET_HIGH_PERFORMANCE=1 (~450 Mbps on en0) |
| local path | models/lfm2.5-24b-a2b/LFM2.5-24B-A2B-QAD-Q4_0.gguf (with the repo README.md) |
| asked by | Alan: "yes, follow new guidelines for benchmarking though" |
The card says this Q4_0 is the QAD checkpoint, and that each GGUF carries the recommended sampling as general.sampling.* metadata.
How it was served
Machine: a Mac Studio (Mac13,1): Apple M1 Max, 64 GB.
| label | dates (UTC) | binary and version | ctx / -np / KV / -fa / --jinja / --reasoning-format | profile | why this label exists |
|---|---|---|---|---|---|
lfm2.5-24b-a2b-qad-q4_0-np4-b11846 | 2026-10-23 21:51–22:29Z | llama.cpp-b11846/build/bin/llama-server, build 11846 (1a24df163) | -c 65536 -np 4 (16,384 per slot) / q4_0 K and V / -fa on / --jinja / default / -ngl 999 --cache-ram 0 | extraction | the first pre-registered plan, 4 suites in flight. Not the measurement of record: 16K per request is below Hermes's 64K |
lfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846 | 2026-10-23 22:37–23:44Z, paused; resumed 2026-10-24 02:03–02:21Z | the same binary | -c 65536 -np 1 (server log: n_slots = 1, n_ctx_slot = 65536) / KV default f16 / -fa on / --jinja / default / -ngl 999 --cache-ram 0 | extraction | the measurement of record. Alan's single-slot 64K rule, PAR=1, 15 suites, VACATE=0. Paused after 4 suites (plus part of enrichment) so Flint could run first ("once that's done kick of flint"); resumed under the same label, and the harness skipped completed cases |
lfm2.5-24b-a2b-qad-q4_0-np1-b11846 (context ladder) | 2026-10-24 09:08–09:09Z | the same binary | -c 8192 … 131072 -np 1 -fa on --jinja -ngl 99, KV default f16 | — | eval/ctx_ladder.py with LADDER_BIN, beside the full fleet |
Seats held: none.
Decisions
- 2026-10-23: picked from the vendor's GGUF set as the QAD Q4_0 (see How it was made).
- 2026-10-24: context ladder run beside the fleet (Alan: "Both, vacate for Flint"; LFM2.5 didn't need the seat).
What went wrong
- Resumed suites under-report in their logs. After the pause and resume, each resumed suite's log printed only the post-resume cases: "0/0 passed" for tagger, tagger_nullable, needle and grounding, and "14/20" for enrichment, which had 34 of its 54 cases done before the pause.
- The result files hold every case, with no duplicates. Scores come from the files.
- A log-derived table after any resume is wrong. Recorded in the lab's benchmarking runbook.
- Run-log
llama_buildandserver_flagswill be wrong for both labels when the run log is backfilled. The truth is build 11846, with the flags under How it was served. A harness version bump is on the source-fix list. - shapegen and sitebuild: harness failures (DatasetDrift;
KeyError: 'nesting'), also seen on sakura 2026-10-11. Excluded from the np1 run. The np4 run's empty sitebuild file is skipped by the bundle generator. - np4 needle: 55 of 110 rejected with HTTP 400 (32K cases against 16K slots). np1 measured all 110.
- The ladder's KV column reads "?": b11846's server log no longer matches
ctx_ladder.py's KV-size regex. KV is taken from RSS growth instead.
What happened to it
Downloaded 2026-10-23 → np4 gauntlet 2026-10-23 → np1 gauntlet of record 2026-10-24 → context ladder 2026-10-24 (to native 128K, 15.3 GB) → decide/1 and edgefill pending. Re-downloadable from the repo and revision above.