FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Qwen3-Next-80B-A3B-Instruct REAP · file← Qwen3-Next-80B-A3B-Instruct REAP

MLX mixed_3_6

Our mixed MLX copy, labelled 4-bit but mostly 3-bit. The one we used most. Made here.

Size
28.0 GB
Status
Deleted 10 Sep 2026
Scored runs
11
Smallest Mac
48 GB estimate, 8K context
Results11 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
contractqwen3-next-80b-reap-mlx-mixed3654.2%40.3–67.4Passed 26/48rapid-mlxextraction2026-09-08
nullschemareap-60b-2026091047.2%36.1–58.6Passed 34/72rapid-mlxextraction2026-09-10
screeningqwen3-next-80b-reap-mlx-mixed3698.3%91.1–99.7Passed 59/60rapid-mlxextraction2026-09-08
synthesisqwen3-next-80b-reap-mlx-mixed36100.0%94.0–100.0Passed 60/60mlx_lmextraction2026-09-09
synthesisreap-60b81.7%70.1–89.4Passed 49/60mlx_lmextraction2026-09-09
synthesisreap-60b-2026091083.3%72.0–90.7Passed 50/60rapid-mlxextraction2026-09-10
toolcallqwen3-next-80b-reap-mlx-mixed3678.3%66.4–86.9Passed 47/60rapid-mlxextraction2026-09-09
toolcallreap-60b-2026091078.3%66.4–86.9Passed 47/60rapid-mlxextraction2026-09-10
toolscalereap-60b-2026091095.0%83.5–98.6Passed 38/40rapid-mlxextraction2026-09-10
worldviewreap-60b47.5%32.9–62.5Passed 19/40mlx_lmextraction2026-09-09
worldviewreap-60b-2026091047.5%32.9–62.5Passed 19/40rapid-mlxextraction2026-09-10

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Corrections2 to the record published 21 October 2026

  • 2 runs voided (the contract run of 8 September 2026; the screening run of 8 September 2026). Every request failed (HTTP 404: the label was sent as the model id), so nothing was measured.

    Corrected 24 October 2026 · a mechanical fix

The recordHow it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

Fileqwen3-next-80b-reap-mlx-mixed36, ~28 GB
FormatMLX mixed_3_6 (4-bit base, 542 overrides, 487 at 3-bit), made here
Where fromsame, 2026-09-08
Statusdeleted 2026-09-10 23:58Z
Hugging Facenot on HF

How it was made

  • Input: the Samsung safetensors (Qwen3-Next-80B-A3B-Instruct REAP safetensors source (SamsungSAILMontreal)), first copy.
  • Command: eval/reap_mlx.sh: mlx_lm convert --hf-path reap-src --mlx-path qwen3-next-80b-reap-mlx-mixed36 -q --q-bits 4 --q-group-size 64 --quant-predicate mixed_3_6, mlx-venv (older mlx). Done by 2026-09-08 20:06Z.
  • Size: ~28 GB (28.1 GB). Two shard sizes were recorded, 5,221,725,661 and 5,353,985,969 bytes; the full shard list and total bytes are not on record.
  • What the header hides: it declares bits: 4, but the config carries 542 per-layer overrides: 487 tensors at 3-bit (including embed_tokens) and 55 at 6-bit, so the build is predominantly 3-bit. The counts were read from the config at 23:53:30Z on 2026-09-10, minutes before deletion. The override map itself was never written down.
  • Who asked: the assistant chose mixed_3_6 unprompted as the "more interesting target"; Alan's 19:02Z "So reap mlx is next then" approved the pipeline.

How it was served

labeldates (UTC)binary and versionctx / -np / KV / -fa / --jinja / --reasoning-formatharness profilewhy this label exists
reapmix36-rapid-mlx2026-09-08 20:08:15Z (finish 21:08 BST)rapid-mlx 0.13.4n/aextractionfirst pass; label used as the wire id, 0 measured (HTTP 404)
qwen3-next-80b-reap-mlx-mixed36screening 2026-09-08 20:38:11Z; contract/2 20:50:38Z; toolcall/1 2026-09-08 23:01:59Z; synthesis/1 2026-09-09 16:51:56Zrapid-mlx 0.13.4 serve <path> --no-spec-decode (no parser flags on the recorded lines); synthesis/1 on mlx_lm 0.31.3wire id = model pathextractionhead-to-head against the IQ4_XS and the 4-bit build
reap-60b2026-09-09 17:39Z synthesis/2, 18:26Z worldview/1 (finish 18:39, 19:26 BST)mlx_lm server --model <path>, 0.31.3n/aextractionthe REAP slot in the role-sized fleet
reap-60b-202609102026-09-10 12:56:54Z to 13:06:04Zrapid-mlx 0.13.4: serve <path> --served-model-name <path> --enable-auto-tool-choice --tool-call-parser qwen3 --reasoning-parser qwen3n/aextractionthe 2026-09-10 "miss REAP" re-bench on the day's exact configuration
(oMLX pool alias)2026-09-09 09:18Z to 12:20ZoMLX 0.6.4n/an/aconsolidation experiment; the toolcall row it produced sits under the IQ4_XS label and is voided

Seats held (all informal): an on-demand accuracy seat in the 2026-09-09 plan; the REAP slot of the role-sized fleet from 2026-09-09 16:45Z until the Metal OOM at 18:43Z; the swap-seat occupant for Hermes/dsh trials 2026-09-09 22:04Z to 2026-09-10 ~15:15Z; the seat behind labs-control.sh reap.

Decisions

  • Why made: GLM mixed_3_6 beat plain 4-bit; expected to be "the smallest and fastest thing on your board holding the extraction seat".
  • Why kept after the 2026-09-09 sweep: the 28 GB build was the surviving MLX REAP and the on-demand accuracy seat.
  • What it taught: it was first read as a REAP success on tool use (78.3) and later as a failure on abstention (47.2); the 2026-09-10/11 reconversion showed the override map moved nullschema +37.5 and toolcall -53 in opposite directions.
  • Why deleted: Alan, 2026-09-10 23:54:33Z, "I don’t need mixed, just the 4 bit quant for mlx and high is what I thought we’d done"; deleted 23:58:19Z after repointing hermes-use.sh, labs-control.sh and the dsh settings to the new 4-bit build.

What went wrong

  • Wrong wire id (404s) on the first pass; re-run with the path as id.
  • oMLX contamination. Served by oMLX it scored toolcall 21/60 = 35.0 against 47/60 on rapid-mlx; availability 12/12 became 0/12. Row voided.
  • Parser false zero. Via dsh and rapid-mlx without --tool-call-parser qwen3 --reasoning-parser qwen3 it returned tool calls as literal text; REAP looked tool-blind for four turns on 2026-09-09 evening. The toolcall/1 suite scored 78.3 on 2026-09-08 with no recorded parser flags and again, exactly 47/60, with them on 2026-09-10, so the suite is insensitive to the flag even though the harnesses were not.
  • Metal OOM. 2026-09-09 18:43Z: kIOGPUCommandBufferCallbackError, 18.4 GB swap; the process kept answering /v1/models while unable to infer, a /v1/models health check cannot tell loaded from working.
  • The 2026-09-10 mix-up (family doc): this build's 47.2 nullschema was first blamed on rapid-mlx not enforcing the schema, which the sentinel probe disproved, then compared with a 95.8 that belongs to a different model.
  • Override map lost. Deleted before the map was written down: its scores are on record, its recipe is not.
  • Standing checks. Partial runs: none (rows match). Stale build stamps: none flagged; rapid-mlx 0.13.4 and mlx_lm 0.31.3 are stamped. Missing flags: the 2026-09-08 rapid-mlx lines have no tool or reasoning parser.

What happened to it

Converted 2026-09-08 ~20:00Z → benched 20:07Z → oMLX pool 2026-09-09 → the fleet → rapid-mlx swap seat → deleted 2026-09-10 23:58:19Z (28.1 GB). No RESTORE entry for the build (RESTORE-deleted-20260911.sh mentions it as unrecoverable). Re-creatable only in principle: source plus --quant-predicate mixed_3_6, but mlx_lm's predicate output was not recorded and the override map cannot be matched.