Model · measured only← Models
lfm2.5-24b-a2b-qad-q4_0-np4-b11846
15 recorded runs. This model has no written entry yet.
- Scored runs
- 15
- Run log entries
- 15
- File
- 13.6 GB
- Needs
- 13.9 GB measured, 64K context
- Smallest Mac
- 24 GB
RecordIdentifier
lfm2.5-24b-a2b-qad-q4_0-np4-b11846
Every result below is as recorded. Without a written entry there is no stated reason for the test and no verdict.
SetupHow it was run
Reference
Kept for reference. It isn't the measurement of record.
The run’s own note: Not the measurement of record: 4 slots give 16K per request, below the 64K Hermes needs. shapegen and sitebuild failed in the harness (dataset lock / dataset field), unscored. needle: 55 of 110 cases rejected (32K cases exceed the 16K slot).
- Server
- 4 slots, 16K of context per request, KV cache q4_0, flash attention on, 4 suites at a time
- Runtime
- llama.cpp b11846 (1a24df163), plain upstream
- Speed
- decode 32.6 tokens/s per request, prefill 583.8 tokens/s, medians. 4 requests in flight, full fleet resident (contended).
- Peak memory
- 13 GB resident, with 64K of context allocated
- Weights
- 13.6 GB file · sha256
d1c03f73ecaf6101… - Coherence check
- not recorded
- Left out by plan
- none
- Machine
- Mac Studio (M1 Max, 64 GB)
- Released
- 24 October 2026, bundle
2026-10-24-977b4a5a
Recorded runs15 scored · 95% intervals
| Suite | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| chainwalk | 50.0% | 9.5–90.5 | Passed 1/2 | llama.cppextraction | 2026-10-23 |
| contract | 72.9% | 59.0–83.4 | Passed 35/48 | llama.cppextraction | 2026-10-23 |
| enrichment | 38.9% | 27.0–52.2 | Passed 21/54 | llama.cppextraction | 2026-10-23 |
| epistemics | 75.0% | 61.2–85.1 | Passed 36/48 | llama.cppextraction | 2026-10-23 |
| grounding | 71.2% | 62.7–78.4 | Passed 89/125 | llama.cppextraction | 2026-10-23 |
| needle50% coverage | 56.4% | 43.3–68.6 | Passed 31/55 | llama.cppextraction | 2026-10-23 |
| nullschema | 38.9% | 28.5–50.4 | Passed 28/72 | llama.cppextraction | 2026-10-23 |
| screening | 48.3% | 36.2–60.7 | Passed 29/60 | llama.cppextraction | 2026-10-23 |
| synthesis | 70.0% | 57.5–80.1 | Passed 42/60 | llama.cppextraction | 2026-10-23 |
| tagger | 62.5% | 56.2–68.4 | Passed 150/240 | llama.cppextraction | 2026-10-23 |
| tagger_nullable | 45.4% | 39.2–51.7 | Passed 109/240 | llama.cppextraction | 2026-10-23 |
| toolcall | 100.0% | 94.0–100.0 | Passed 60/60 | llama.cppextraction | 2026-10-23 |
| tools | 86.7% | 62.1–96.3 | Passed 13/15 | llama.cppextraction | 2026-10-23 |
| toolscale | 100.0% | 91.2–100.0 | Passed 40/40 | llama.cppextraction | 2026-10-23 |
| worldview | 27.5% | 16.1–42.8 | Passed 11/40 | llama.cppextraction | 2026-10-23 |
Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.
Every run log entry for this model (15)
| Finished | Suite | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| chainwalkchainwalk/3 | llama.cppextraction | 50.0% | Coverage 2/2 | g f4bfcf6f040b0978d 129a9e74408c8919c 1aa3b1ae71955ea1 | |
| enrichmentenrichment/3 | llama.cppextraction | 38.9% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc d5bc927e67bb57cc | |
| taggertagger/2 | llama.cppextraction | 62.5% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac a483f88f4f3e2b34 | |
| screeningscreening/1 | llama.cppextraction | 48.3% | Coverage 60/60 | g 0661b49a10be2d2ad 64c97ca2269c29afc 99bb653fb40dd11b | |
| contractcontract/2 | llama.cppextraction | 72.9% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c 66489f7c9323f5c1 | |
| toolstools/1 | llama.cppextraction | 86.7% | Coverage 15/15 | g 2763675a5c3da564d 33a44812f0ad4c39c 260296041a390e31 | |
| toolscaletoolscale/1 | llama.cppextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c a8f3c0fe0f501532 | |
| toolcalltoolcall/1 | llama.cppextraction | 100.0% | Coverage 60/60 | g 804e8fc9457649d8d 10af88c170513155c 924b8542372c5a6e | |
| worldviewworldview/2 | llama.cppextraction | 27.5% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c 268f16986415e52a | |
| nullschemanullschema/1 | llama.cppextraction | 38.9% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc 10a03dd6e47e9d1b | |
| synthesissynthesis/2 | llama.cppextraction | 70.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 89397895ec36c956 | |
| tagger_nullabletagger-nullable/1 | llama.cppextraction | 45.4% | Coverage 240/240 | g 04e875431e26c4b5d db3addf23a2bc4cec 260eec9c0d3b6ebc | |
| epistemicsepistemics/4 | llama.cppextraction | 75.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 39d2305b9089ea3b | |
| groundinggrounding/1 | llama.cppextraction | 71.2% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 8a1861d31a5df2fd | |
| needleneedle/1 | llama.cppextraction | 56.4% | Coverage 50% coverage | g c38dec33aa1c86c2d 5c5199b9eee8a595c 0d0138cc0d658e0d |