FreshContext A context lab · Fife , home
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Granite Guardian 4.1 8B · file ← Granite Guardian 4.1 8B

Q4_0

The one we kept, and our judge now. Made here from IBM's file, for speed. Requantised here from IBM's Q4_K_M.

Size
4.8 GB
Status
On disk
Scored runs
8
Smallest Mac
24 GB measured, 64K context
Results 8 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
screening granite-guardian-4.1-8b-b10893-control-nocoh88.3%77.8–94.2Passed 53/60 llama.cpp extraction2026-10-11
screening granite-guardian-4.1-8b-b11552-nocoh90.0%79.9–95.3Passed 54/60 llama.cpp extraction2026-10-11
screening granite-guardian-4.1-8b-q4_0-np1-c64k-b1184695.0%86.3–98.3Passed 57/60 llama.cpp extraction2026-10-26
screening granite-guardian-4.1-8b-q4_0-np1-c64k-kvq4-b1184690.0%79.9–95.3Passed 54/60 llama.cpp extraction2026-10-26
screening granite-guardian-4.1-8b-q4_0-np1-c8k-kvq4-b1184690.0%79.9–95.3Passed 54/60 llama.cpp extraction2026-10-26
screening_native granite-guardian-4.1-8b-q4_0-np1-c64k-b1184690.0%79.9–95.3Passed 54/60 llama.cpp extraction2026-10-26
screening_native granite-guardian-4.1-8b-q4_0-np1-c64k-kvq4-b1184688.3%77.8–94.2Passed 53/60 llama.cpp extraction2026-10-26
screening_native granite-guardian-4.1-8b-q4_0-np1-c8k-kvq4-b1184688.3%77.8–94.2Passed 53/60 llama.cpp extraction2026-10-26

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Setup How it was run

granite-guardian-4.1-8b-q4_0-np1-c64k-b11846

Of record

The measurement of record for this model.

The run’s own note: The live SCREEN seat's Q4_0 file (requantised here from IBM's Q4_K_M), single slot at 64K, f16 KV. A judge model: only screening/1 and screening-native/1 run; the generation suites say nothing about its job. llama-bench (contended): pp512 382.3, tg128 52.9 t/s.

Server
1 slot, 64K of context per request, KV cache f16 (default), flash attention on, 1 suite at a time
Runtime
llama.cpp b11846 (1a24df163), plain upstream
Speed
decode 51.4 tokens/s, prefill 339.6 tokens/s, medians. Single stream, full fleet resident (contended); short judge prompts and verdicts.
Peak memory
14.65 GB resident, with 64K of context allocated
Weights
4.8 GB file · sha256 d9bdbcc561ade73e…
Coherence check
not run (judge model: the gate asks for prose, which a judge does not write)
Left out by plan
tagger, tagger_nullable, needle, grounding, enrichment, epistemics, synthesis, worldview, nullschema, toolcall, toolcall2, toolscale, tools, contract, chainwalk, shapegen, sitebuild
Machine
Mac Studio (M1 Max, 64 GB)
Released
26 October 2026, bundle 2026-10-26-1faf1f39

granite-guardian-4.1-8b-q4_0-np1-c64k-kvq4-b11846

Reference

Kept for reference. It isn't the measurement of record.

The run’s own note: Control run: the same file and build with a q4_0 KV cache, the only change from the run of record. q4_0 KV turns 5 verdicts into prose with no JSON on screening/1.

Server
1 slot, 64K of context per request, KV cache q4_0, flash attention on, 1 suite at a time
Runtime
llama.cpp b11846 (1a24df163), plain upstream
Speed
decode 47.7 tokens/s, prefill 337.5 tokens/s, medians. Single stream, full fleet resident (contended); short judge prompts and verdicts.
Peak memory
7.46 GB resident, with 64K of context allocated
Weights
4.8 GB file · sha256 d9bdbcc561ade73e…
Coherence check
not run (judge model: the gate asks for prose, which a judge does not write)
Left out by plan
tagger, tagger_nullable, needle, grounding, enrichment, epistemics, synthesis, worldview, nullschema, toolcall, toolcall2, toolscale, tools, contract, chainwalk, shapegen, sitebuild
Machine
Mac Studio (M1 Max, 64 GB)
Released
26 October 2026, bundle 2026-10-26-fca5e15e

granite-guardian-4.1-8b-q4_0-np1-c8k-kvq4-b11846

Reference

Kept for reference. It isn't the measurement of record.

The run’s own note: Control run at the live seat's context and KV (8K, q4_0) on b11846. Identical results to the 64K q4_0 run, so context size is not the cause.

Server
1 slot, 8K of context per request, KV cache q4_0, flash attention on, 1 suite at a time
Runtime
llama.cpp b11846 (1a24df163), plain upstream
Speed
decode 47.7 tokens/s, prefill 337.5 tokens/s, medians. Single stream, full fleet resident (contended); short judge prompts and verdicts.
Peak memory
4.99 GB resident, with 8K of context allocated
Weights
4.8 GB file · sha256 d9bdbcc561ade73e…
Coherence check
not run (judge model: the gate asks for prose, which a judge does not write)
Left out by plan
tagger, tagger_nullable, needle, grounding, enrichment, epistemics, synthesis, worldview, nullschema, toolcall, toolcall2, toolscale, tools, contract, chainwalk, shapegen, sitebuild
Machine
Mac Studio (M1 Max, 64 GB)
Released
26 October 2026, bundle 2026-10-26-b5187dee
The record How it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

Filegranite-guardian-4.1-8b-Q4_0.gguf (made as _q40test/granite-guardian-4.1-8b-Q4_0.gguf)
FormatQ4_0 GGUF, 4,824,740,928 B
Where frommade here 2026-09-11 13:16Z by llama-quantize --allow-requantize from the Q4_K_M (two rounding passes)
Statuson disk; the live SCREEN seat

How it was made

Q=llama.cpp/build/bin/llama-quantize
$Q --allow-requantize granite-guardian-4.1-8b-Q4_K_M.gguf \
models/_q40test/granite-guardian-4.1-8b-Q4_0.gguf Q4_0

Output: "quant size = 4597.83 MiB (4.60 BPW)", 5.86 s.

  • Tool: llama.cpp llama-quantize, binaries built 2026-09-10 13:19 local, reporting b10893 c32d1dabe (tree HEAD is the local diffusion commit 63cbb9227 on top) — reconstructed from the binary mtime and llama-server --version, 2026-10-23; the quantize run itself did not print a version.
  • Date: 2026-09-11 13:16Z (file mtime "11 Sep 14:16" local).
  • Size / hash: 4,824,740,928 B; sha256 d9bdbcc561ade73eaed56fb60b4c66ee4e2ebaa43f3aedaca6c392a93093a78a (both computed 2026-10-23).
  • Who asked / why: Alan, "what would you quant to?". The assistant picked Guardian as "the cleanest candidate … requantising it to Q4_0 isolates format from architecture completely". It was a speed experiment; the assistant said at the time that quality "must be tested before you act on it".
  • Moved 2026-09-11 15:51Z from _q40test/ to granite-guardian-4.1-8b/granite-guardian-4.1-8b-Q4_0.gguf.

How it was served

label / roledates (UTC)binary and versionctx / -np / KV / -fa / --jinja / --reasoning-formatharness profilewhy this label exists
llama-bench only2026-09-11 13:16Z, 14:17Zllama.cpp/build/bin/llama-bench (b10893)-p 0 -n 64 -ngl 99 -r 2/3—the Q4_0 lever test
SCREEN seat (no run-log label)2026-09-20 19:24Z → present.local/bin/llama-server → llama.cpp b10893 c32d1dabe--jinja -c 8192 -np 1 -ctk q4_0 -ctv q4_0 -fa on -ngl 999 --alias guardian; no reasoning format—Alan's "what happened with guardian model"; under launchd from 2026-09-29 (com.freshcontext.seat-guardian)
granite-guardian-4.1-8b-b10893-control-nocoh2026-10-11 ~10:57Z–11:07Zb10893 c32d1dabe (/props: b10893-c32d1dabe, n_ctx 8192, 1 slot, ftype Q4_0, template sha fed2756d2d24e127)seat flags as aboveextraction, --no-coherencelive-build control for the b11552 upgrade gate (manifest)
granite-guardian-4.1-8b-b11552-nocoh2026-10-11 ~10:57Z–11:08Zllama.cpp-b11552/build/bin/llama-server, b11552 23b0202a1--jinja -c 8192 -np 1 -ctk q4_0 -ctv q4_0 -fa on -ngl 999 --alias guardianextraction, --no-coherencethe new build under test; the run without --no-coherence refused to score

Fleetview roster entry: granite-guardian-4.1-8b, upstream model guardian, label "IBM Granite Guardian 4.1 8B Q4_0", note "SCREENER — 100% (20/20/20) · llama.cpp ONLY (36.7% on oMLX) · 8K". It is not in fleet-models.ini.

Decisions

  • Made to prove the Q4_0 speed lever on one model, same weights; Alan: "we have a lever, chase it, measure it, prove it".
  • Kept 2026-09-11 15:14Z in the "KEEPING (Q4_0 builds, drafters, working MTP)" list and moved into the model dir.
  • Became the only Guardian GGUF 2026-09-12 when its parent was deleted under "Q4_K_M with a Q4_0 sibling".
  • Seated 2026-09-20 with the parent's 100% in its roster note. That afternoon the assistant had accepted "it gets seated on llama.cpp or re-benched, never assumed". It was seated on llama.cpp; it was not re-benched.
  • 2026-10-11 gate: 88.3/90.0 called "identical" across builds; recommendation to switch to b11552. Switch not executed: the symlink still points at b10893 (2026-10-23).
  • Not published. A 2026-10-07 inventory ranked the 2026-09-11 Q4_0s as "double-quantised, low value to publish".

What went wrong

  • Seated on a borrowed score. The fleetview note, a lab note's SCREEN row and later fleet tables (e.g. "It scored 20/20 on each of keep, fabricated and misattributed") quote 100% (20/20/20) for this file. Its own runs are 20/16/17 and 20/17/17.
  • The coherence precondition refused the judge (2026-10-11). Probes like "reply with the single word" got <think>\n</think>\n<score> no </score>. Membership 67% < 85%, instruction-following 67% < 83%, and the harness said "Retire the artefact or fix the quantisation; do not file the numbers". The runs were then made with --no-coherence. The assistant later logged it as a harness false refusal on specialist models. Whether the Q4_0's format drift and the probe failures share a cause is not measured.
  • The live seat was used as a benchmark arm (2026-10-11): the b10893 control ran against the production seat.
  • Smoke tests show tag looping: '<think>\n</think>\n<score> no </score></think>\n<score>' on a 16-token budget. The Q4_K_M did the same on 2026-08-31, so this is the model's interface, not the requant.
  • Missing-flag check: --jinja present; no --reasoning-format. Guardian's think block reaches content, and that is how the grader reads it.
  • Stale build stamps: none. Run-log runtime_version comes from /props and matches the binaries.

What happened to it

Requantised here 2026-09-11 13:16Z → speed-benched 2026-09-11 → kept as sole GGUF 2026-09-12 → seated 2026-09-20 → launchd 2026-09-29 → first quality runs 2026-10-11 (88.3 / 90.0) → re-measured 2026-10-26 on b11846 (95.0 with f16 KV; 90.0 with q4_0 KV, the cause of the malformed verdicts) → on disk and serving at models/granite-guardian-4.1-8b/granite-guardian-4.1-8b-Q4_0.gguf (4,824,740,928 B, checked 2026-10-23). Not on HF. Recreating it needs IBM's Q4_K_M plus the command above. A single-pass Q4_0 from the BF16 in the HF cache would be a different artefact.