FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

DiffusionGemma 26B-A4B · file← DiffusionGemma 26B-A4B

MLX 4-bit

The build that ran: mlx-community's 4-bit MLX copy, scored on four tests. Deleted on 11 October. Mlx-community.

Size
16.5 GB
Status
Deleted 11 Oct 2026
Scored runs
4
Smallest Mac
32 GB estimate, 8K context
Results4 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
synthesisdiffusiongemma-26b-mlx80.0%68.2–88.2Passed 48/60mlx-vlmextraction2026-09-11
toolcalldiffusiongemma-26b-mlx5.0%1.7–13.7Passed 3/60mlx-vlmextraction2026-09-11
toolscalediffusiongemma-26b-mlx100.0%91.2–100.0Passed 40/40mlx-vlmextraction2026-09-11
worldviewdiffusiongemma-26b-mlx77.5%62.5–87.7Passed 31/40mlx-vlmextraction2026-09-11

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

The recordHow it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

Filemodels/diffusiongemma-26b-mlx-4bit/ (4 safetensors shards)
FormatMLX 4-bit (affine; 8-bit overrides per the 2026-10-07 inventory), 16,543,055,405 B by the HF listing
Where fromdownloaded 2026-09-11 00:10Z–~00:32Z from mlx-community/diffusiongemma-26B-A4B-it-4bit with a curl loop; local tokenizer_config.json hand-patched 2026-09-11 10:52Z
Statusdeleted 2026-10-11
Hugging Facehttps://huggingface.co/mlx-community/diffusiongemma-26B-A4B-it-4bit

Where it came from

  • Repo: mlx-community/diffusiongemma-26B-A4B-it-4bit, main, revision not pinned.
  • Command: a one-off download script, run with uv's CPython 3.12 at 2026-09-11 00:10:33Z: reads https://huggingface.co/api/models/<repo>?blobs=true, then curl -sL each *.safetensors|*.json|*.txt|*.jinja|*.model from resolve/main/ into models/diffusiongemma-26b-mlx-4bit. Its docstring records why not the GGUF: unsloth #6336 (garbled output on Metal).
  • Complete: by 2026-09-11 00:32Z ("diffusiongemma-26b-mlx-4bit 15 GB ✓").
  • Modified locally: 2026-09-11 10:52Z, for the OptiQ attempt, tokenizer_config.json was rewritten (extra_special_tokens list ["<|video|>"] → dict {"video_token": "<|video|>"}) with a .bak-<timestamp> copy left beside it. The board runs (08:24–08:35Z) predate this edit, so they ran on the file as downloaded.
  • Who asked: Alan, "Can you pause reap and start diffusion?" and, for the MLX build specifically, "Grab it".

How it was served

label / roledates (UTC)binary and versionctx / -np / KV / -fa / --jinja / reasoningharness profilewhy this label exists
diffusiongemma-26b-mlx2026-09-11 08:24Z–08:35Zdflash-venv/bin/python -m mlx_vlm.server --model models/diffusiongemma-26b-mlx-4bit; mlx-vlm 0.6.17 in dflash-venv (listed 00:33Z the same night, not stamped at run time); mlx 0.32.2 per the run-log runtime_versionsserver defaults (no --max-tokens, no KV flags); diffusion canvas/steps at mlx-vlm defaults, not recorded; schema enforcement unsupportedextraction (temperature 0, seed 42, max_tokens 2048, thinking off), --model-id = the full local path, --no-resumethe only board run of a diffusion model; one label for the whole board
speed probe (no label)2026-09-11 07:49Z–07:54ZMacPaw Fast-dLLM-mlx via uv run (Python 3.13, fetched its own mlx-vlm/mlx-lm)--steps 16 --threshold 0.9 --no-think --use-self-conditioning --max-canvas-length 64 --max-new-tokens 256 --warmup; classic path --max-new-tokens 256—throughput measurement
OptiQ attempt (no label)2026-09-11 10:20Z–10:53Zoptiq serve (mlx-optiq 0.5.6; transformers 5.12.1 then pinned 4.57.6)——second MLX server; never answered

The server needed the full path as the request's model field; a short name made it look the id up on HF and 404. Seats held: none. Not in fleet-models.ini, fleetview or launchd.

Decisions

  • Downloaded instead of the GGUF after unsloth #6336.
  • Benched the same morning.
  • Not seated. Alan: "synthesis is fine, in fact fast synthesis / planning helper may actually be really strong" then "no wiring just now, we will have models to try out".
  • Kept at the 2026-09-11 disk triage as "the one diffusion model that actually works at a usable speed", and again on 2026-09-16 ("tested, no faster build on disk KEEP").
  • Deleted 2026-10-11 by the gpt-oss/diffusion sweep: "toolcall 5% (3/60) disqualifies it for any seat".
  • Not published (a third party's conversion; nothing to publish).

What went wrong

  • nullschema refused, not run. RuntimeError: could not verify schema enforcement against <server>: HTTPError: HTTP Error 500 at 08:32Z; the sentinel probe returned "Structured response_format is not supported with diffusion models." The gate worked as designed; cost: the pre-registered test never ran.
  • Markdown-fenced JSON. Responses wrapped JSON in ``` `json ``` fences; flagged as a possible grader cost, and it did not cost worldview.
  • Left loaded. It stayed resident after its board finished; at 08:53Z six models were loaded and swap reached ~36–37 GB before it was shed.
  • tokenizer_config.json modified in place for the OptiQ attempt; the deleted folder was therefore not byte-identical to the HF repo.
  • Wrong re-download id. RESTORE-deleted-20261011.sh says hf download velaia/diffusiongemma-26b-mlx-4bit --revision main; that repo returned HTTP 401 on 2026-10-11 and an audit called it "made up". The script, untracked in git on 2026-10-23, was not corrected. Correct source: mlx-community/diffusiongemma-26B-A4B-it-4bit.
  • Standing checks: partial runs — none (row counts match); stale build stamps — none (llama_build empty); missing flags — not applicable (not llama.cpp).

What happened to it

Downloaded 2026-09-11 00:10Z–00:32Z → speed-probed 07:49Z → benched 08:24Z–08:35Z → OptiQ attempt failed 10:53Z → kept at the 2026-09-11 and 2026-09-16 triages → deleted 2026-10-11 (RESTORE entry exists in RESTORE-deleted-20261011.sh, with the wrong repo id) → re-downloadable from mlx-community/diffusiongemma-26B-A4B-it-4bit (current main may differ from what was fetched on 2026-09-11).