FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Model family← Models

DiffusionGemma 26B-A4B

Google's diffusion model, which writes a block of text over several passes instead of one word at a time. It wrote well here, but ran at ordinary speed and almost never reached for a tool.

Files tested
2
Scored runs
4

Made by Google

OverviewWhy we tested it · what happened · should you use it

How this page was made. The tests run automatically on our Mac Studio, and every score here comes straight from those runs. This overview was written by Claude, an AI model, from the lab's own records: the run log, notes kept at the time, and the working sessions where each decision was made. Quotes are Alan's words, with spelling fixed. How these pages are made.

Why we tested it

On the night of 10 September Alan asked "Can we run mercury 2.5?", a diffusion model only offered as an online service. Our agents pointed to DiffusionGemma, an open diffusion model with builds for a Mac. Its published speeds, over 1,000 tokens a second, were all on Nvidia's data-centre GPUs, and twice our agents concluded from its design, without testing it, that there was nothing worth running. Alan: "I think you should try it cause even that would be faster tokens".

There was a second question: would a model that writes this way fail differently when asked for a value that isn't in the document?

What it took

More than usual. llama.cpp, which runs most of our board, didn't know this design, so the GGUF never loaded. We switched to an MLX build served by mlx-vlm, and tried three other ways to run it, mostly for speed. When our agents called the options dead ends, Alan pushed back: "don’t accept defeat, keep looking online, world view."

What happened

  • It wasn't fast here. On the Mac Studio, an M1 Max, it managed 28.6 tokens a second on its quickest path, and 20.8 on mlx-vlm's standard one: about the speed of an ordinary word-by-word model. Another figure the benchmark tool printed suggested six times that, but it counted passes over the block, not text produced.
  • It did well on three of our tests. It scored 80% on turning tool results into an answer, 77.5% on noticing when evidence is missing, conflicting or out of date, and 100% on choosing between tools.
  • It almost never reached for a tool. On our test of whether a model calls a tool when it needs one, it scored 5%, 3 of 60. The server passed the tools through; the model rarely saw the need.
  • The second question went unanswered. mlx-vlm can't hold a diffusion model to a set output format, so our harness refused to run the test of saying "I don't know" rather than publish a number it couldn't trust.
  • It was never put to work. Alan saw a use: "synthesis is fine, in fact fast synthesis / planning helper may actually be really strong", but chose no wiring for now. The GGUF was deleted on 11 September and the MLX build on 11 October, with its 5% tool score as the reason.

What it taught us

  • Diffusion speed figures don't carry over to a Mac yet. Separate tests of other diffusion models that day found they are limited by computing power on Apple Silicon.
  • Choosing the right tool and noticing you need one are different skills. Tool perception is binary.
  • Measure rather than reason from the design. Our agents twice ruled it out on paper; measuring showed it ran, and wrote well.

Should you use it?

For writing and summarising on a Mac, it did well, at ordinary speed. Not for tool use, and not where you need output in a fixed format, since the server we used can't enforce one. llama.cpp couldn't load it when we tried. Check the smallest Mac for the file below.

The files2 tested · each has its own page
  • MLX 4-bit

    The build that ran: mlx-community's 4-bit MLX copy, scored on four tests. Deleted on 11 October.

    Mlx-community · Deleted 11 Oct 2026 · 16.5 GB · 4 scored runs · smallest Mac 32 GB

  • GGUF Q4_K_M

    A GGUF that llama.cpp couldn't load. Downloaded twice, never run, deleted on 11 September.

    Unsloth · Deleted 11 Sep 2026 · 16.8 GB · never scored

The lab's recordEvery decision, with its date · the model's specification
Decisions, in order
  • 2026-09-10 23:43Z — measure rather than reason. Alan: "I think you should try it cause even that would be faster tokens". Assistant: "Fair challenge — I've been reasoning from architecture rather than measuring".
  • 23:44Z — GGUF first, for comparability. The assistant picked unsloth Q4_K_M over the 15.4 GB MLX build because "it goes on the same runtime as your whole board", so a run gives "a comparable row rather than another MLX-vs-llama.cpp confound".
  • 2026-09-11 00:09Z — switch to MLX. Stopped the GGUF on unsloth issue #6336; downloaded mlx-community/diffusiongemma-26B-A4B-it-4bit instead. Alan: "Well this is an exiting twist in direction and probably more future facing".
  • 07:49Z — first generation. Alan: "Try it". Fast-dLLM-mlx answered 42 correctly.
  • 07:59Z — re-download the GGUF as a "third arm", after the assistant found issue #24529's M3 Max numbers and retracted its dismissal.
  • 08:23Z — put it on the board. Served on mlx_vlm.server because Fast-dLLM-mlx has no server. Pre-registered in the script header: "standard masked diffusion does not revise committed tokens... So this should NOT show a different failure profile on nullschema's lure arms. If it beats K2's 16.7% or Qwen3.8's 58.3% materially, the model of why is wrong." The script also states the runtime confound "up front". Outcome: the prediction could not be tested — nullschema was refused.
  • 08:35Z — verdict. "a capable writer and summariser, at good speed, that can't be constrained and won't reach for tools... not the agentic seat". Alan, 08:36Z: "synthesis is fine, in fact fast synthesis / planning helper may actually be really strong". The assistant proposed the Hermes triage_specifier and kanban_decomposer slots. Alan, 08:39Z: "no wiring just now, we will have models to try out". It was never seated.
  • 10:14Z — GGUF arm closed. Arch diffusion-gemma not in the build.
  • 10:20–10:53Z — OptiQ attempt. optiq serve recognised "DiffusionGemma: vendored masked-diffusion decode" and loaded it (RSS 15.9 GB) but failed in mlx_lm's server on transformers 5.12.1 (eos_token_ids), then on transformers 4.57.6 on a list-valued extra_special_tokens, then on has_chat_template, which the assistant traced to a bug in mlx-optiq 0.5.6. Stopped after "one more attempt". The OptiQ "tool-call healing" layer was flagged as something that would inflate toolcall if it had worked.
  • 13:50Z — keep the MLX, drop the GGUF. Disk triage: the GGUF in Tier 1 ("needs a fork you don't have"), the MLX "the one diffusion model that actually works at a usable speed" kept. Alan, 15:13Z: "kill what you can disk wise, keep any q4_0 and drafters..."; GGUF removed 15:14Z.
  • 2026-10-11 — MLX build deleted. Sweep by another of our agents, recorded in a lab note, which quotes Alan as "gpt-oss and diffusion models can go"; that line is not otherwise on record. Stated reason: "toolcall 5% (3/60) disqualifies it for any seat".
What we learned, in the lab's words
  • It runs on an M1 Max, at autoregressive-MoE speed, not diffusion speed. Fast-dLLM-mlx path (steps 16, threshold 0.9, 64-token canvas, self-conditioning): 28.6 tok/s effective, 9.66 s per 256-token canvas, mean 25.9 steps; mlx-vlm classic path: 20.8 tok/s, 12.0 steps. The benchmark's own work/s column (172.6 and 232.7) is canvas × steps, not output; quoting it would claim a 6× speedup that does not exist. Probe, not a run-log entry.
  • Competitive where it could be measured, disqualified where it could not. Board: worldview 77.5 (tied qwen3.8-27b), synthesis 80.0, toolscale 100, toolcall 5.0 (table below).
  • Tool selection and tool perception separate again. toolcall 3/60 with finish_reason stop 57 / tool_calls 3, so the server does carry tools and the model "rarely perceives the need"; toolscale 40/40. Same split as qwen3-8b, spark-x2.5-4b and lfm2.5-8b-a1b, now across autoregressive dense, MoE and block diffusion (Tool perception is binary and tracks the checkpoint, not size or vendor claims).
  • No constrained decoding on this runtime. mlx_vlm.server answers "Structured response_format is not supported with diffusion models."; plain generation works. The harness's schema gate refused nullschema rather than produce an unenforced number, so the abstention/lure question this model was brought in for was never measured. grounding, which also needs a schema, was never attempted.
  • Worldview arms: 31/40 with single_source 0/8 and superseded 8/8 (re-tallied 2026-10-05). It cites, but it does not qualify single-source claims.
  • The GGUF path was never available here. The Q4_K_M declares diffusion-gemma, which upstream llama.cpp does not register; the assistant attributed support to an unmerged PR (#24427) or the unsloth fork. The M3 Max figures of 28→6 tok/s degrading and 48–61 tok/s after a local Metal patch are reported in llama.cpp issue #24529, not measured here.
  • A bug report was over-generalised, then corrected. The assistant first stopped the GGUF download on unsloth issue #6336 (garbled Metal output, M5 Pro, llama.cpp b9632), then noted the different chip and build; the GGUF later turned out not to load at all, so that bug never applied.
  • Context for the site: the same day's other diffusion work (LLaDA-MoE, LLaDA-8B) produced Diffusion LLMs are compute-bound on Apple Silicon, so quantising them buys nothing and the fc/diffusion-confidence llama.cpp branch; those are LLaDA results, not DiffusionGemma ones. A lab note records the diffusion question as its starting point.
Why it was tested, in the lab's words
  • It began with a different model. Alan, 2026-09-10 23:40Z, while the assistant was busy: "Can we run mercury 2.5?" then "Am guessing it needs Vida", "Cuda". Mercury 2.5 is API-only, so the assistant pointed at DiffusionGemma as the one open diffusion model with Metal-reachable builds and said "Not tonight".
  • Alan, 23:42Z: "Research it - world view". After reading the vendor guide the assistant again concluded "nothing to run here".
  • Alan, 23:43Z: "I think you should try it cause even that would be faster tokens". The assistant conceded it had been "reasoning from architecture rather than measuring".
  • Alan, recorded inside the next assistant turn: "Just to be clear that I'm not being annoying, the ones that beat mercury will be worth watching hard, if they hit numbers like that in benchmarking then that's amazing".
  • Alan, 23:50Z: "Can you pause reap and start diffusion?"; 2026-09-11 00:10Z, on the switch from GGUF to MLX: "Well this is an exiting twist in direction and probably more future facing"; 00:11Z "Grab it".
  • The assistant's framing of what it would answer, written into the plan at 00:10Z: "token revision changes what failure looks like"; a good score on the lure arms, where everything else sits at 12–37%, would be "a finding about architecture rather than about a model".
  • Alan, 07:50Z, after the assistant had called three runtimes dead ends: "But you’re not getting off the hook too easily because this is being used in the wild, don’t accept defeat, keep looking online, world view.", and 07:59Z "Yes dig deeper though".
  • Alan, 08:24Z, on what the result was for: "honestly, I'd be happy to prove out fast then move on to what they're actually able to handle, no models are big enough to be great at most stuff BUT it's a sign of things to come".
  • Compared against (same five suites, from the 2026-09-10 table printed in the bench script): k2-36b, qwen3.8-27b, gemma4-12b. It is the first and only diffusion model on the board.
Open questions
  • Does the lure/abstention profile differ for a diffusion decoder? Untested: needs a runtime that enforces a JSON schema on a diffusion model (llama.cpp with diffusion-gemma support, if it lands upstream).
  • Is the 28.6 tok/s (Fast-dLLM) or the mlx-vlm classic path (20.8) what mlx_vlm.server used for the board? The server path was not timed; the board ran at the server's defaults.
  • Would the upstream GGUF path reproduce issue #24529's 48–61 tok/s on an M1 Max? Not measured.
  • Licence and upstream revision were never recorded.
The model in detail
fieldvalue
vendor / upstreamGoogle, google/diffusiongemma-26B-A4B-it (instruction-tuned). Community builds seen 2026-09-10: mlx-community/diffusiongemma-26B-A4B-it-4bit (plus 5-bit and 6-bit), unsloth/diffusiongemma-26B-A4B-it-GGUF (BF16, Q4_K_M, Q5_K_M, Q6_K, Q8_0), RedHatAI/nvidia FP8 and NVFP4
revisionnot on record (both download scripts fetched resolve/main without a pinned revision)
licencenot on record (no licence was read during the download or the run)
release datenot on record
architectureblock-diffusion (masked-diffusion) mixture-of-experts. Parameter counts as recorded: "25.2B parameters (sparse MoE, ~3.8B active per step)" per diffusiongemma.org; the assistant then read the repo name as 26B total / 4B active ("the same as K2-36B"). Layer, head and expert counts were not read on this box
GGUF architecture stringgeneral.architecture = diffusion-gemma (read from the local Q4_K_M header); upstream llama.cpp only knows dream, llada, llada-moe
MLX configmodel_type diffusion_gemma; an inventory on 2026-10-07 recorded transformers 5.8.0.dev0 in the config and 236 per-layer overrides to 8-bit (reconstructed from a sub-agent report, not re-read)
vendor speed claimsH100 1,008 tok/s, H200 1,288 tok/s (FP8 checkpoint); the vendor guide says CPU-only GGUF inference "sacrifices the speed that is the model's whole point"
runtimes tried heremlx_vlm.server (served the board); MacPaw Fast-dLLM-mlx and mlx-vlm's classic DiffusionGemma path (speed probes only); optiq serve (mlx-optiq 0.5.6, failed); mmastrac/diffgemma Rust/Metal (built, device probe only, uses its own weight pack); llama.cpp could not load the GGUF