Model · measured only← Models
lfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846
15 recorded runs. This model has no written entry yet.
- Scored runs
- 15
- Run log entries
- 15
- File
- 13.6 GB
- Needs
- 13.9 GB measured, 64K context
- Smallest Mac
- 24 GB
RecordIdentifier
lfm2.5-24b-a2b-qad-q4_0-np1-c64k-b11846
Every result below is as recorded. Without a written entry there is no stated reason for the test and no verdict.
SetupHow it was run
Of record
The measurement of record for this model.
The run’s own note: Single slot at 64K (benchmark rule). Paused after 4 suites and resumed under the same label; scored from result files. screening below chance: the model emits tool-call syntax where no tools are offered (audited; harness sound).
- Server
- 1 slot, 64K of context per request, KV cache f16 (default), flash attention on, 1 suite at a time
- Runtime
- llama.cpp b11846 (1a24df163), plain upstream
- Speed
- decode 102.5 tokens/s, prefill 763.3 tokens/s, medians. Single stream, full fleet resident (contended).
- Peak memory
- 13.9 GB resident, with 64K of context allocated
- Weights
- 13.6 GB file · sha256
d1c03f73ecaf6101… - Coherence check
- passed
- Left out by plan
- shapegen, sitebuild
- Machine
- Mac Studio (M1 Max, 64 GB)
- Released
- 24 October 2026, bundle
2026-10-24-12403c71
Recorded runs15 scored · 95% intervals
| Suite | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| chainwalk | 50.0% | 9.5–90.5 | Passed 1/2 | llama.cppextraction | 2026-10-24 |
| contract | 75.0% | 61.2–85.1 | Passed 36/48 | llama.cppextraction | 2026-10-24 |
| enrichment | 51.9% | 38.9–64.6 | Passed 28/54 | llama.cppextraction | 2026-10-24 |
| epistemics | 70.8% | 56.8–81.8 | Passed 34/48 | llama.cppextraction | 2026-10-24 |
| grounding | 72.8% | 64.4–79.8 | Passed 91/125 | llama.cppextraction | 2026-10-24 |
| needle | 54.5% | 45.2–63.5 | Passed 60/110 | llama.cppextraction | 2026-10-24 |
| nullschema | 44.4% | 33.5–55.9 | Passed 32/72 | llama.cppextraction | 2026-10-24 |
| screening | 25.0% | 15.8–37.2 | Passed 15/60 | llama.cppextraction | 2026-10-24 |
| synthesis | 75.0% | 62.8–84.2 | Passed 45/60 | llama.cppextraction | 2026-10-24 |
| tagger | 62.1% | 55.8–68.0 | Passed 149/240 | llama.cppextraction | 2026-10-23 |
| tagger_nullable | 50.0% | 43.7–56.3 | Passed 120/240 | llama.cppextraction | 2026-10-23 |
| toolcall | 100.0% | 94.0–100.0 | Passed 60/60 | llama.cppextraction | 2026-10-24 |
| tools | 86.7% | 62.1–96.3 | Passed 13/15 | llama.cppextraction | 2026-10-24 |
| toolscale | 100.0% | 91.2–100.0 | Passed 40/40 | llama.cppextraction | 2026-10-24 |
| worldview | 55.0% | 39.8–69.3 | Passed 22/40 | llama.cppextraction | 2026-10-24 |
Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.
Every run log entry for this model (15)
| Finished | Suite | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| chainwalkchainwalk/3 | llama.cppextraction | 50.0% | Coverage 2/2 | g f4bfcf6f040b0978d 129a9e74408c8919c e7c8396d9ed03203 | |
| contractcontract/2 | llama.cppextraction | 75.0% | Coverage 48/48 | g cc2d23c5bae45474d 1fe53d30cb5480f6c d983a1eab242a2c5 | |
| screeningscreening/1 | llama.cppextraction | 25.0% | Coverage 60/60 | g 0661b49a10be2d2ad 64c97ca2269c29afc b74d2f0fd98f854b | |
| toolstools/1 | llama.cppextraction | 86.7% | Coverage 15/15 | g 2763675a5c3da564d 33a44812f0ad4c39c 11f1bcab9b59dbbb | |
| toolscaletoolscale/1 | llama.cppextraction | 100.0% | Coverage 40/40 | g a2bc3751abfa28e0d b24abb108c409178c dc258c69347efb48 | |
| toolcalltoolcall/1 | llama.cppextraction | 100.0% | Coverage 60/60 | g 804e8fc9457649d8d 10af88c170513155c 81b819c35899e092 | |
| nullschemanullschema/1 | llama.cppextraction | 44.4% | Coverage 72/72 | g d7448e0f5fc3f9afd 3b91280bb40372efc fcb6d784f212ea90 | |
| worldviewworldview/2 | llama.cppextraction | 55.0% | Coverage 40/40 | g 9acfba1b704feb28d e97835de9e6fc227c ecefd0dc821be544 | |
| synthesissynthesis/2 | llama.cppextraction | 75.0% | Coverage 60/60 | g 4156dd3c13361e8bd 267d1954e225c5b6c 3b8b41b1a21cb97b | |
| epistemicsepistemics/4 | llama.cppextraction | 70.8% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc e79f2ead72c51e1f | |
| enrichmentenrichment/3 | llama.cppextraction | 51.9% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc ce56e16d2c0e729c | |
| groundinggrounding/1 | llama.cppextraction | 72.8% | Coverage 125/125 | g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 2ba60c7c54da5b59 | |
| needleneedle/1 | llama.cppextraction | 54.5% | Coverage 110/110 | g c38dec33aa1c86c2d 5c5199b9eee8a595c 65a31ff3750251f7 | |
| tagger_nullabletagger-nullable/1 | llama.cppextraction | 50.0% | Coverage 240/240 | g 04e875431e26c4b5d db3addf23a2bc4cec 36c31b70aa553f7e | |
| taggertagger/2 | llama.cppextraction | 62.1% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 1fe2084ade408d11 |