Model family← Models
Underdog-Saluki-27B-1.0
ConwayResearch's 2-bit build of Qwen3.8-27B, in a 7.9 GB file. It kept the 4-bit model's tool use but lost most of its ability to say "I don't know", and on our Mac Studio it was slow.
- Files tested
- 2
- Scored runs
- 10
- Smallest Mac
- 16 GB for the file we kept
Made by ConwayResearch · Apache 2.0
How this page was made. The tests run automatically on our Mac Studio, and every score here comes straight from those runs. This overview was written by Claude, an AI model, from the lab's own records: the run log, notes kept at the time, and the working sessions where each decision was made. Quotes are Alan's words, with spelling fixed. How these pages are made.
Why we tested it
Alan, on 11 October: "there's another model that interests me vs bonsai - underdog saluki looks interesting". Saluki is Qwen3.8-27B squeezed to about 2 bits per weight, built by ConwayResearch from ISTA-DASLab's compressed version of it.
It was meant to answer two questions. We already had the same model at 4 bits and at 1 bit, as Ternary-Bonsai-2-27B, so Saluki would show what 2 bits costs. And Bonsai 2 needs a special fork of llama.cpp, while Saluki runs on the standard one, so it might replace it. We expected little beyond that.
What it took
It runs on standard llama.cpp as it comes. The trouble was speed. Alan left it running: "you can run the gauntlet, i'm going out for a couple hrs". With four requests at a time and our other models on the Mac Studio, it read about 96 tokens a second and wrote between 2.6 and 5.2, far below the 15 to 25 we had guessed. After 33 minutes we stopped the full set of tests, which would have taken most of a night, and ran only the seven where both comparison builds already had scores.
What happened
- Tool use survived 2 bits. 100% on our tool test and on picking the right tool from a growing set, the same as at 4 bits. Its makers say it beats the full model at tool calling; here all three builds scored 100%, so no difference could show.
- So did turning tool results into an answer, 83.3% as at 4 bits, and it scored 95.0% at checking extractions.
- Admitting gaps did not. It declined 29.2% of the time when the answer wasn't there, against 69.4% to 76.4% at 4 bits, and finding and citing answers in a document fell to 68.0% from 80.0%. Both are level with the 1-bit build, at 31.9% and 66.4%: on these tests 2 bits bought the 1-bit model's quality, not the 4-bit model's.
- Not one to run all day on the Mac Studio. It might still replace Bonsai 2 on the M4 Pro Mac mini at equal quality and without the special fork, if it is fast enough there. That hasn't been measured.
Only the seven comparison tests ran to the end. Both files are still on disk.
What it taught us
- Compression doesn't cost every skill equally. At 2 bits it kept tool use and lost most of its ability to say "I don't know".
- A speed taken with other work on the machine isn't a speed figure. We still have no clean single-request number for it.
Should you use it?
If you want Qwen3.8-27B's tool use in a 7.9 GB file, it keeps it. Don't use it where answers must be grounded or gaps admitted: if your Mac has room, the 4-bit Qwen3.8-27B was far better at admitting gaps here, and better at grounding. Files compressed this way need more computing per token than 4-bit ones, and on our M1 Max it was slow, though we never timed it on its own. Check the smallest Mac for the file we kept, below.
- IQ2-mixkept
The one we kept: the model itself, squeezed to about 2 bits. Tested against its 4-bit and 1-bit siblings, never seated.
ConwayResearch · On disk · 7.9 GB · 10 scored runs · smallest Mac 16 GB
- mmproj Q8_0
The add-on for reading images. Downloaded with the model, never loaded.
Vision projector · On disk · 0.6 GB · never scored
Decisions, in order
- 2026-10-11 12:23Z — assessed from the card. Pre-registered: "Speed is the open question on this box"; "2-bit i-quants take more compute to decode than Q4_0", so "expect it below what a Q4_0 of the same size would do".
- 2026-10-11 12:55Z — download on Alan's "grab saluki then". Queued behind the Sakura download.
- 2026-10-11 13:53Z — unattended run. Alan: "you can run the gauntlet, i'm going out for a couple hrs". Saluki became part 2 of
unattended.sh, beside the full fleet. - 2026-10-11 15:26Z — cut to a subset by the assistant while Alan was out: 33 minutes in, decode speed put the full gauntlet at "most of a night". It was restarted on
grounding nullschema synthesis worldview toolcall toolscale screening, "the seven suites where both comparison builds of the same base model already have scores" (SUMMARY.md). - 2026-10-11 16:03Z — verdict: "Not a Mac Studio seat. As a bonsai2 replacement on the M4 Pro it would remove the prism fork at equal quality... but only if the M4 Pro's speed is acceptable... Low priority either way" (SUMMARY.md).
- Not seated; not deleted. No fleetview, fleet-models.ini or launchd reference (grep 2026-10-23). Both files remain on disk.
What we learned, in the lab's words
- Tools, synthesis and screening survive 2-bit; abstention and grounding do not. nullschema 29.2 and grounding 68.0, "level with the 1-bit bonsai2 — so on this box's suites, 2-bit buys bonsai2's quality, not the 4-bit model's" (SUMMARY.md).
- The card's "beats the original at tool calling" is a tie here: all three compressions hit 100 on toolcall (SUMMARY.md).
- Dense 2-bit i-quants are compute-bound on the M1 Max: ~96 t/s prefill and 2.6–5.2 t/s decode per request with four in flight, beside the full fleet (server log, SUMMARY.md). The assistant had guessed "roughly 15–25 t/s here" beforehand. No clean single-stream number exists.
The compression ladder (same base)
| suite | Qwen3.8 Q4_0 (4-bit) | Saluki IQ2-mix (2-bit) | bonsai2 PTQ1_0 (1-bit) |
|---|---|---|---|
| toolcall | 100 | 100 | 100 |
| toolscale | 100 | 100 | — |
| synthesis | 83.3 | 83.3 | 83.3 |
| screening | 93.3 (MLX 4-bit) | 95.0 | 98.3 |
| worldview | 80.0 | 75.0 | 82.5 |
| grounding | 80.0 (MLX 4-bit: 92.0) | 68.0 (85/125) | 66.4 |
| nullschema | 69.4–76.4 | 29.2 | 31.9 |
(Quoted from the run's SUMMARY.md, written 2026-10-11 16:03Z. The Saluki column matches the run log; the comparison columns are other labels' run-log rows from various dates and builds, per the summary's own caveat.)
Why it was tested, in the lab's words
- Alan, 2026-10-11 12:22Z: "there's another model that interests me vs bonsai - underdog saluki looks interesting". Then "what do you think we'll gain from the model over what we have".
- The assistant's answer: "On [the Mac Studio], probably not much. Its one real gain is replacing bonsai2", the one model forcing the prism fork. Plus "A cheap answer to what compression costs", though "it's knowledge, not a capability gain".
- Alan, 12:53Z, about the Sakura download: "grab the sakura version if it fits, always handy having models to test without fucking around". Then "where we at with saluki?" and "grab saluki then".
- Compared against:
qwen3.8-27bQ4_0 and bonsai2 PTQ1_0 (same base), plus the earlier GSQ-RCO 3-bit buildqwen38-gsqrco-iq3s.
Open questions
- Single-stream decode speed on the Mac Studio (llama-bench), and on the M4 Pro, which decides the bonsai2 replacement (SUMMARY.md).
- The full gauntlet (tagger, tagger_nullable, needle, enrichment, epistemics, contract, chainwalk, tools) was never completed.
The model in detail
| field | value |
|---|---|
| vendor | ConwayResearch ("Built by Underdog") |
| upstream repo | ConwayResearch/Underdog-Saluki-27B-1.0 (108,900 downloads on 2026-10-11) |
| revision | not on record (resolve/main); file sha256 4a673518b11b1c4445f9b9a9d3c40356f5ba39f6a6260dcb25f3fbfd475d9efb |
| licence | Apache 2.0 |
| released | not on record; press coverage dated 2026-10-09 (MarkTechPost link) |
| base / derivation | base Qwen3.8-27B; card says it was derived from Qwen3.8-27B-GSQ-RCO-GGUF by ISTA-DASLab |
| architecture | Qwen3.8-27B dense (no GGUF header read) |
| format | IQ2-mix; run log / configs: IQ2_XS - 2.3125 bpw (IQ2-mix) |
| vendor claims | tool calling 88 vs 84 (full Qwen3.8); BFCL v4 parallel 42/100 vs 35; IFEval 93.5 vs 91.5; IFBench 72.7 vs 71.0; SWE-bench Verified 30/50 vs 33; MBPP+ 78.0 vs 83.9; "~82-85% of full model's competition math"; "~20% of parallel-call responses have minor formatting issues" |
| vendor serving advice | llama-server -m... --jinja -ngl 99 -fa on -c 32768 |