Q4_0

How it was made

• Input: models/k2-horizon-36b-a4b/K2-Horizon-36B-A4B-Q4_K_M.gguf (22,368,011,616 B), itself made here from IFM's BF16 (see K2-Horizon-36B-A4B Q4_K_M (quantised here from IFM's BF16)). This file has therefore been rounded twice (BF16→Q4_K_M→Q4_0). The assistant noted this at the time: "Every Q4_0 measured today came from --allow-requantize off an ALREADY QUANTISED file, so it carries two rounding passes".
• Command:

llama.cpp-k2horizon/build/bin/llama-quantize --allow-requantize \
K2-Horizon-36B-A4B-Q4_K_M.gguf models/_q40test/K2-Horizon-36B-A4B-Q4_K_M__Q4_0.gguf Q4_0

The fork's own quantize and bench were used so the format comparison stays inside one binary. Upstream llama-quantize had just failed with unknown model architecture: 'k2-horizon'.

• Tool: k2horizon fork @ 35999d1 (build 20).
• Date: 2026-09-11, ~13:22Z (file mtime "11 Sep 14:22" local, ls -la models/k2-horizon-36b-a4b/ on 2026-10-23).
• Size: 21,280,078,176 B (stat, 2026-10-23). sha256 not on record.
• Who asked / why: Alan: "we have a lever, chase it, measure it, prove it". The lever was Q4_0's speed over Q4_K_M at equal bit-width on Metal, first seen on other models that afternoon. K2-36B was one of five models in the sweep.
• Renamed 2026-09-11 15:51Z to k2-horizon-36b-a4b/K2-Horizon-36B-A4B-Q4_0.gguf, because the old name "claims both formats in one filename", a provenance trap.

How it was served

Seats held (outside the run log):

• Hermes / agent seat, 2026-09-11 15:53Z → 2026-09-12 09:52Z: the seat via swap-to-36b.sh (--jinja --reasoning-format deepseek -c 131072 -np 1 -fa on -ctk q8_0 -ctv q8_0 -ngl 999 --alias k2-36b). On 2026-09-11 it was briefly at 32768, then 65536 ("Hermes minimum"), then back to 131072.
• Seat-race candidate, 2026-09-18/19:
• A server -c 16384 -np 1 -fa on --jinja --reasoning-format deepseek for the edgefill probe;
• A server with SEAT_EFFORT=none for the fleet table;
• A server -c 32768 -np 1 -ctk q4_0 -ctv q4_0 -fa on --jinja --reasoning-format deepseek, router default reasoning_effort: none.
• swap-seat occupant, 2026-09-20 → present: fleetview BENCH["k2-36b"], fork binary, -c 32768 -np 1 --reasoning-format deepseek --alias k2-36b plus SEAT_COMMON --jinja -ctk q4_0 -ctv q4_0 -fa on -ngl 999.

Run-log runtime: llama.cpp b20 (35999d1) (runtime_source "props"). It was not in fleet-models.ini or any launchd plist on 2026-10-23.

Decisions

• Made to price the Q4_0 speed lever on an MoE.
• Kept and made the serving build 2026-09-11 15:51Z. It was the only one of three K2 Q4_0 builds with a full board. Assistant: "k2-36b stays the default — 100% tool perception and now 49.7 tok/s". Alan: "you can do 1 - 5".
• Survived the 2026-09-12 sweep as the Q4_0 sibling that justified deleting the Q4_K_M.
• Not chosen as driver 2026-09-19. Alan: "ornith slightly faster but I suspect k2 a better model" and "I think ornith has to stay for kv cache and context lengths". Delisted with the note "Strong at schema/synthesis; resurrect for that".
• Kept as a swap-seat occupant from 2026-09-20 (fleetview.py).
• Not published to HF. No reason is on record; it is a requant of a requant, so a clean single-pass build would be the candidate.

What went wrong

• Double rounding. It is a requant of a local Q4_K_M, not of BF16 or Q8_0. The single-pass alternative (K2-Horizon-36B-A4B Q4_0 (requantised here from the local Q8_0, one extra rounding pass), from the Q8_0) reached 49.48 tok/s, but its quality run failed twice at the harness schema probe and it was deleted unscored.
• Misleading filename K2-Horizon-36B-A4B-Q4_K_M__Q4_0.gguf. The run manifests record that path, which no longer exists after the rename, so automated matching of labels to files failed later.
• All-empty toolcall responses (2026-09-11 13:26Z). All 60 responses were the empty string and still passed. On inspection they were real tool calls carried in tool_calls (outcome: "called"), so the result stands.
• Metal OOM / Hermes "Compute error" 2026-09-11 ~16:02Z: served at 128k beside three other seats. Restarted at 32k. fleetview.py also records that "k2-36b answered 'Compute error' until the roster was trimmed".
• Race UI defaulted to k2-36b 2026-09-19. Alan: "I suspect your model race is always loading k2 36b, getting exactly 31tps every time". A 30-second refresh "silently re-armed the default race pair and made every race look like k2-36b". It was fixed at 01:01Z. Race readings for k2-36b taken before the fix are suspect.
• Hermes "Compute error" (Alan pasted the error, model: k2-36b error: Compute error., at 16:06Z) — see the OOM entry below.
• Orphaned think tag 2026-09-18. </ifm|think> leaked into content because the template has three think tags (<ifm|think>, <ifm|think_fast>, <ifm|think_faster>) and the parser tracks one. An edgefill score of 5% was a serialisation artefact, not capability.
• Missing flag (standing check). The board server was started without --jinja. Whether build 20 applies the chat template by default is not on record. The toolcall run made real tool calls (outcome: "called"), which suggests the template was in effect. Every later serving line for this file has --jinja.
• Partial runs (standing check): none. All five results files have the full case count.
• Wrong "not on disk" claim. Finding Byte alignment, not bit count, governs decode speed (revised 2026-09-14) says k2-36b is no longer on disk and its architecture is not recorded anywhere. This file was on disk throughout (ls 2026-10-23), and the architecture is in the lab's session record.

What happened to it

Requantised here 2026-09-11 → benched 2026-09-11 → serving build 2026-09-11 → off the Hermes seat 2026-09-12 → seat-race candidate, delisted 2026-09-19 → swap-seat occupant 2026-09-20 → on disk at models/k2-horizon-36b-a4b/K2-Horizon-36B-A4B-Q4_0.gguf (21,280,078,176 B, checked 2026-10-23). Not on HF. Re-creatable only via BF16 → Q4_K_M → Q4_0 with the two commands in this doc and in K2-Horizon-36B-A4B Q4_K_M (quantised here from IFM's BF16). A single-pass BF16→Q4_0 would be a different artefact.