Devstral Small 2 24B · file← Devstral Small 2 24B

UD-Q4_K_XL

The file every score came from. A 4-bit copy, chosen so our tool tests would measure the model rather than its speed. Deleted in a tidy-up. Unsloth.

Size: 14.5 GB
Status: Deleted 7 Sep 2026
Scored runs: 3
Smallest Mac: 32 GB estimate, 8K context

Results3 scored · 95% intervals

Suite	Score	95% interval	Passed	Runtime · profile	Recorded
groundingdevstral-small-2-24b-b10618	79.2%	71.3–85.4	Passed 99/125	llama.cppextraction	2026-08-25
nullschemadevstral-small-2-24b-b10618	62.5%	51.0–72.8	Passed 45/72	llama.cppextraction	2026-08-25
screeningdevstral-small-2-24b-b10618	100.0%	94.0–100.0	Passed 60/60	llama.cppextraction	2026-08-25

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

The recordHow it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

File	`Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf`
Format	UD-Q4_K_XL GGUF, 14,506,151,072 B
Where from	downloaded 2026-08-24, sha256 verified against the LFS oid
Status	deleted 2026-09-07 17:36Z
Hugging Face	https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF

Where it came from

Repo / file: unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF / Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf.
sha256: b44e34b78180fc3ab1abbe1edad9f1f3926fdca10eed3bfae168b065e683f6cd, checked after assembly by a one-off script. The script's "OK" line for this file is not in the evidence excerpt; that it passed is implied by the script writing .dl_complete only after both files verified, and by the Q6_K (second in the script) being present at 17:43 BST.
Revision: not on record (resolve/main).
Date: 2026-08-24, between 16:15Z (download restarted) and 16:43Z.
Method: six parallel range requests with --speed-limit 51200 --speed-time 30, after a single-stream hf_hub_download crawled at ~22 Mbps.
Who asked: Alan, "let's try out Devstral Small 2, what sizes does it come in?". The quant choice was the assistant's: "UD-Q4_K_XL at 14.5GB should decode in the 20–30 t/s range, which is enough to complete scenarios".

How it was served

label	dates (UTC)	binary and version	ctx / -np / KV / -fa / --jinja / --reasoning-format	harness profile	why this label exists
`devstral-small-2-24b-b10618`	2026-08-25 08:09Z (server up) – 08:37Z (last run-log row)	`llama.cpp/build/bin/llama-server`, b10618 (`eb25b7263`)	`-c 32768 -np 4 -cb -ngl 99`; KV default (f16); `-fa` not set; `--jinja` yes; `--reasoning-format` not set	extraction (thinking off, max_tokens 2048)	first measurable build; the `-b10618` suffix keeps it from mixing with b10369 numbers
(not in run log) Toolery 143, same server	2026-08-25 08:09Z–~08:30Z	b10618	as above	Toolery `--adapter raw --tier all --trials 1 --concurrency 4`	same fleet re-run, Toolery stage
(not in run log) AgentDojo banking + slack	2026-08-25 20:43Z → finished by 2026-08-26 08:47Z	b10618	`-c 32768 -np 4 -cb -ngl 99 --jinja`	AgentDojo `--model VLLM_PARSED`, utility then `--attack important_instructions`, logdir `runs_full/devstral-small-2-24b`	the injection-resistance question
(no label; gate abort)	2026-08-24 ~19:2xZ	b10369 (`6e62ba538`)	`-c 32768 -np 4 -cb -ngl 99 --jinja`	—	a one-off script; aborted at the tool-call gate
(no label; gate abort)	2026-08-24 19:31Z	b10369	same	—	a one-off script with a GPU lock; aborted at the gate

Run-log runtime for the scored label: b10618 (eb25b7263), runtime_source "evidence". The run-log llama_build stamp agrees. The endpoint is confirmed by all three result manifests. Never a seat, a router preset or a launchd plist.

Decisions

Downloaded as the fleet-comparable 4-bit.
Measured first in the b10618 re-run, because it and ornith were the reason for the rebuild.
Not seated. "Right now it doesn't have a confirmed role". After AgentDojo, "12b is still the pick, and Devstral's role remains unproven". Named screener "if screening proves necessary" (a lab note §7).
Deleted 2026-09-07 17:36Z as the smaller of two "duplicate quants", keeping the Q6_K, on Alan's "Go for it". The record does not show that the deletion weighed which file carried the scores.
Published on the site as devstral-small-2-24b-b10618 (status "candidate").

What went wrong

Tool-call gate aborts on b10369 (2026-08-24). Both gates failed with JSONDecodeError on "arguments":"{"location": "Warsaw"}". They were diagnosed as a llama-server serialisation bug, "model-specific — Gemma and Qwen serialise fine through the identical path". That diagnosis is the basis of the line "llama.cpp build 10369 emitted tool-call arguments with unescaped quotes, making Devstral and Ornith unmeasurable" (a lab note §5). What the record does not settle (reconstructed):
both b10369 gates piped the response through zsh print;
the next morning, the same print-based probe failed on b10618 for devstral, ornith and gemma4-e4b;
the cause turned out to be print itself, which "interprets backslash escapes, so it mangled \"";
the file-based probe then passed on b10618;
b10369 was never re-probed without print, so whether it was really broken for Devstral is not on record.

Cost: one evening, and a full fleet re-run on a new build. Fix: gates write to a file.

Scored file deleted, unscored sibling kept (2026-09-07). Afterwards, inventories reported devstral as an untested Q6_K: "no result file, no seat UNTESTED" and "NEVER BENCHMARKED". Fixed in model_inventory.py by token-set matching.

What happened to it

Downloaded 2026-08-24 → gate-blocked on b10369 2026-08-24 → benched on b10618 2026-08-25 (run log, Toolery, AgentDojo) → never seated → deleted 2026-09-07 17:36Z (duplicate-quant pass). No RESTORE entry (no RESTORE-deleted-20260907.sh exists; grep -i devstral over the RESTORE scripts finds only the 2026-10-11 Q6_K entry). Re-downloadable from unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF; verify against sha256 b44e34b7…83f6cd.