Squish INT3

A 3-bit MLX copy made with Squish, the MLX side of our August speed comparison. Since gone. MLX, made here.

Size: 3.8 GB
Status: gone by 2026-09-10; deletion not on record
Scored runs: 0

ResultsNever scored

This file never completed a scored run. Its record below says why.

The recordHow it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

File	`models/Qwen3-8B-int3/` (3.8 GB per a lab note)
Format	Squish INT3, "4.0 bpw effective"
Where from	made 2026-08-11/12 with Squish from the BF16 (inferred)
Status	gone by 2026-09-10; deletion not on record

How it was made

Source: the BF16 safetensors (Qwen3-8B BF16 safetensors (mlx-community)) (inferred: Squish converts from safetensors, per a lab note).
Tool / command: Squish (konjoai, squish-ai from PyPI, installed into squish-venv with transformers<5, SQUISH_SKIP_DOCTOR=1). The conversion command is not on record (the 2026-08-11 session is not on record); the INT3 format was a fallback because --int4 needs a squish_quant Rust extension that is not shipped (per a lab note).
Output: models/Qwen3-8B-int3/, "MLX-native ~4 bpw... 3.8 GB" in a lab note and "4.08 GB / 4.0 bpw effective" in the runtime note (3.8 GiB = 4.08 GB, the same figure in two units).
Date: 2026-08-11 or 08-12 (the note is dated 2026-08-11/12). Exact bytes and sha256 not on record.
Who asked: the runtime comparison; Alan's words not on record.

How it was served

label	dates (UTC)	binary and version	ctx / -np / KV / -fa / `--jinja` / `--reasoning-format`	harness profile	why this label exists
none (no run-log label)	2026-08-11/12	raw mlx-lm; Squish (BUSL-1.1) via `mlx_lm.load`; vllm-mlx 0.4.1 with `--continuous-batching --max-num-seqs {1,8,16}`	n/a	none (own throughput scripts: 32 prompts x 200 tokens, temperature 0)	runtime comparison
(same dir, re-run)	`fleet-roster/models-scripts/sq_int3_ab.sh`: two `squish run... --skip-doctor` timings with `SQUISH_SKIP_DOCTOR=1`	Squish	n/a	none	wall-clock A/B of Squish INT3

This one directory supplied the "MLX" side of every runtime row in the lab note: raw mlx-lm 46.0 t/s, Squish 37.6 t/s, vllm-mlx 43.5 / 53.9 / 87.5 t/s at 1 / 8 / 16 sequences. The llama.cpp side used a different file (the Q4_K_M).

Decisions

Built to isolate the runtime variable "at a fraction of the cost" of a 70 GB download (per a lab note).
The conclusion drawn (serve on llama.cpp; MLX only for fine-tuning) was made on a size-confounded comparison and the note itself says so: "MLX won wall-clock... only because its quant was smaller". The matched-bit-width test used gemma-4-26B-A4B.
Alan, 2026-10-04, on Squish: "squish hasn't been used in ages"; squish-venv is retired.

What went wrong

Squish INT4 and INT8 did not work (per a lab note's install trail): squish_quant missing for INT4, an unresolved AutoTokenizer ImportError in the server subprocess for INT8. INT3 was the only Squish format that served.
Undated deletion. models/Qwen3-8B-int3 was MISSING on 2026-09-10 08:28Z and absent from the 2026-09-07 listing. A working session on 2026-10-11 still lists it as a missing name referenced by sq_int3_ab.sh with no RESTORE entry.
Script rot. eval/conc_vllmmlx.py was rewritten on 2026-10-09 (repo paths, git commit e060cfc); the V= line in the repo copy is not valid Python (V="os.path.join(os.path.dirname(sys.executable), "vllm-mlx")"), so the script as stored cannot be run.
Standing checks: not applicable (no run-log entries).

What happened to it

Made 2026-08-11/12 -> throughput comparison -> gone by 2026-09-07 -> no RESTORE entry -> can only be recreated by downloading the BF16 again and re-running Squish (command not on record).