FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Qwen3.8-27B · file← Qwen3.8-27B

mixed 3-bit MLX pack with DFlash2 drafter

A 3-bit pack for chad, a coding tool. Three tests, then deleted: chad has no server. Nathansutton, chad's default.

Size
12.1 GB
Status
Deleted 18 Sep 2026
Scored runs
3
Smallest Mac
24 GB estimate, 8K context
Results3 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
screeningqwen38-27b-q3kxl-chad55% coverage84.8%69.1–93.3Passed 28/33mlx_lmextraction2026-09-18
synthesisqwen38-27b-q3kxl-chad85.0%73.9–91.9Passed 51/60mlx_lmextraction2026-09-18
worldviewqwen38-27b-q3kxl-chad92.5%80.1–97.4Passed 37/40mlx_lmextraction2026-09-18

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

The recordHow it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

Fileqwen38-chad-default/
Formatmixed 3-bit MLX, 12,089,224,041 B plus 1,146,451,617 B drafter
Where fromdownloaded 2026-09-18
Statusdeleted 2026-09-18
Hugging Facehttps://huggingface.co/nathansutton/Qwen3.8-27B-UD-Q3_K_XL-DFlash2-MLX

Where it came from

  • Repo: nathansutton/Qwen3.8-27B-UD-Q3_K_XL-DFlash2-MLX; revision not pinned. Quant text from the manifest: mlx 3-bit g64 base with 498 per-tensor overrides.
  • Date and tool: 2026-09-18 08:09Z huggingface_hub.snapshot_download into models/qwen38-chad-default; stalled, restarted at 08:47Z with 8 workers; served at 08:59Z.
  • The assistant explained that the DFlash2 drafter and fused attention kernel exist only inside chad.

How it was served

labeldates (UTC)binary and versionctx / -np / KV / -fa / --jinja / --reasoning-formatharness profilewhy this label exists
qwen38-27b-q3kxl-chad2026-09-18 09:12–09:23mlx_lm server 0.31.3 from the PonyExl3 venv--model models/qwen38-chad-default, no draft model, no response_format enforcementextractionthe quality question the drafter cannot affect (verification is exact): is the 3-bit pack as good as the 4-bit and the ternary?

Before the suites, harness.coherence returned COHERENT, confidence 0.95, 0% non-Latin, 12 of 12 instruction and world checks. The gauntlet script ran synthesis, worldview, screening, epistemics and enrichment under the default profile. Seats held: none.

Decisions

  • Comparison drawn by the assistant: worldview 92.5 for this 3-bit pack against 82.5 for Bonsai-2 PQ2_0 and 80.0 for Q4_0; synthesis 85.0 against 83.3 and 83.3. The two runs differ in runtime and in whether the schema was enforced (mlx_lm ignores response_format), so the comparison is not like for like.
  • Alan on the tool: "chad is a coding harness, not an llm how the heck did we end up installing it?", "it doesn't even serve over http", "we need to can it, it's not what we need". The assistant accepted the error: it had not checked for a server before installing.
  • Deleted 2026-09-18 11:31Z with qwen38-bonsai2-mlx, bonsai2-dflash2 and qwen38-dflash-patched, from Alan's answer of 'All of it — 40 GB', keeping the gemma4-26b MLX and the chad clone and venv.
  • Not published.

What went wrong

  • Wrong tool chosen without checking (above): about an hour of work on a benchmark number that does not transfer to any seat.
  • Partial screening run (above).
  • Unenforced schemas. mlx_lm ignores response_format, so the run log's schema_enforced false applies; a zero-violation run there is the model obeying.
  • Build stamp: the run log stamps these rows with the llama.cpp build string '0.4.0-dev (build 10893, commit c32d1dabe)' although the server was mlx_lm 0.31.3; the runtime column is correct, the build column is a carried-over stamp.

What happened to it

Downloaded 2026-09-18 08:09–08:59Z → coherence gate and three suites 09:00–09:24Z → deleted 2026-09-18 11:31Z. No RESTORE-deleted-*.sh entry. Re-downloadable from nathansutton/Qwen3.8-27B-UD-Q3_K_XL-DFlash2-MLX; the clone chad (v2.1.0 at commit a4d6588) was kept.