FreshContext A context lab · Fife , home
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Underdog-Saluki-27B-1.0 · file ← Underdog-Saluki-27B-1.0

IQ2-mix

The one we kept: the model itself, squeezed to about 2 bits. Tested against its 4-bit and 1-bit siblings, never seated. ConwayResearch.

Size
7.9 GB
Status
On disk
Scored runs
10
Smallest Mac
16 GB estimate, 8K context
Results 10 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
grounding saluki-iq2mix-np4-b1155268.0%59.4–75.5Passed 85/125 llama.cpp extraction2026-10-11
needle saluki-iq2mix-np4-b11552 void68.2%47.3–83.6Passed 15/22 llama.cpp extraction2026-10-11
nullschema saluki-iq2mix-np4-b1155229.2%19.9–40.5Passed 21/72 llama.cpp extraction2026-10-11
screening saluki-iq2mix-np4-b1155295.0%86.3–98.3Passed 57/60 llama.cpp extraction2026-10-11
synthesis saluki-iq2mix-np4-b1155283.3%72.0–90.7Passed 50/60 llama.cpp extraction2026-10-11
tagger saluki-iq2mix-np4-b11552 void100.0%43.8–100.0Passed 3/3 llama.cpp extraction2026-10-11
tagger_nullable saluki-iq2mix-np4-b11552 void100.0%51.0–100.0Passed 4/4 llama.cpp extraction2026-10-11
toolcall saluki-iq2mix-np4-b11552100.0%94.0–100.0Passed 60/60 llama.cpp extraction2026-10-11
toolscale saluki-iq2mix-np4-b11552100.0%91.2–100.0Passed 40/40 llama.cpp extraction2026-10-11
worldview saluki-iq2mix-np4-b1155275.0%59.8–85.8Passed 30/40 llama.cpp extraction2026-10-11

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Corrections 3 to the record published 21 October 2026
  • On the tagger run of 11 October 2026: voided. Partial file from a stopped gauntlet: 3 of 240 cases were run, and its 100% coverage counts only rows present, so the row is voided.

    Corrected 26 October 2026 · a mechanical fix

  • On the tagger_nullable run of 11 October 2026: voided. Partial file from a stopped gauntlet: 4 of 240 cases were run, and its 100% coverage counts only rows present, so the row is voided.

    Corrected 26 October 2026 · a mechanical fix

  • On the needle run of 11 October 2026: voided. Partial file from a stopped gauntlet: 22 of 110 cases were run, and its 100% coverage counts only rows present, so the row is voided.

    Corrected 26 October 2026 · a mechanical fix

The record How it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

Filesaluki-27b/Underdog-Saluki-27B-1.0-IQ2-mix.gguf
FormatIQ2-mix GGUF, 7,898,369,152 B (stat)
Where fromdownloaded 2026-10-11 from ConwayResearch/Underdog-Saluki-27B-1.0, sha256-verified
Statusbenched; on disk
Hugging Facehttps://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0

Where it came from

  • Repo / file: ConwayResearch/Underdog-Saluki-27B-1.0, Underdog-Saluki-27B-1.0-IQ2-mix.gguf.
  • Command (file on disk):
REPO="https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0/resolve/main"
"$HOME/eval/tools/pardl.sh" "$REPO/$f" "$DST/$f" 6
shasum -a 256 "$f" # compared to 4a673518b11b1c4445f9b9a9d3c40356f5ba39f6a6260dcb25f3fbfd475d9efb
  • Revision: resolve/main, unpinned. The LFS oid/sha256 was recorded from the HF tree API.
  • sha256: 4a673518b11b1c4445f9b9a9d3c40356f5ba39f6a6260dcb25f3fbfd475d9efb, "sha256 OK".
  • Date: 2026-10-11. File mtime 15:00 local = 14:00Z (ls, 2026-10-23).
  • Who asked: Alan, "grab saluki then".

How it was served

labeldates (UTC)binary and versionctx / -np / KV / -fa / --jinja / --reasoning-formatprofilewhy this label exists
saluki-iq2mix-np4-b115522026-10-11 14:52Z → 15:26Z (full gauntlet, stopped); 15:27Z → ~16:02Z (subset)llama.cpp-b11552/build/bin/llama-server, /props b11552 (23b0202a1)-c 65536 -np 4 (16,384 per slot), -ctk q4_0 -ctv q4_0, -fa on, --jinja, no --reasoning-format, -ngl 999 --cache-ram 0extractionbuild-stamped label for the 4/2/1-bit comparison, run via swapseat_gauntlet.sh with VACATE=0 (seats stay up)

No --mmproj on either start, so the vision projector was never loaded. The 64k / 16k-per-slot window is wider than the card's -c 32768. Never seated. Run-log runtime b11552 (23b0202a1) (runtime_source "props").

Decisions

  • Downloaded for the compression ladder and the bonsai2 replacement question.
  • Subset run, a decision made in Alan's absence.
  • Kept on disk, not seated. "Not a Mac Studio seat... Low priority either way" (SUMMARY.md).
  • Not published (third-party file).

What went wrong

  • Stopped gauntlet. The first run (all 17 suites, four in parallel) was killed at 15:26Z for speed. It took several attempts to stop; the final kill loop used STOP/KILL. Cost about 33 minutes. The partial files were left under the same label.
  • Partial rows published as complete (the tagger, tagger_nullable and needle rows). This should be fixed at the source: void them, or set attempted to the suite size.
  • Decode under 4-way load is not a speed measurement. The SUMMARY says so, and no clean number exists.
  • Standing checks: --jinja present; build stamp consistent; partial runs found (3).

What happened to it

Downloaded 2026-10-11 → subset-benched 2026-10-11 → not seated → on disk at models/saluki-27b/Underdog-Saluki-27B-1.0-IQ2-mix.gguf (7,898,369,152 B, checked 2026-10-23). No RESTORE entry needed. Re-downloadable from ConwayResearch/Underdog-Saluki-27B-1.0 (verify the sha256 above).