FreshContext A context lab · Fife , home
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Qwen3-Next-80B-A3B-Instruct REAP · file ← Qwen3-Next-80B-A3B-Instruct REAP

MLX mixed_3_6

Our mixed MLX copy, labelled 4-bit but mostly 3-bit. The one we used most. Made here.

Size
28.0 GB
Status
Deleted 10 Sep 2026
Scored runs
11
Smallest Mac
48 GB estimate, 8K context
Results 11 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
contract qwen3-next-80b-reap-mlx-mixed3654.2%40.3–67.4Passed 26/48 rapid-mlx extraction2026-09-08
nullschema reap-60b-2026091047.2%36.1–58.6Passed 34/72 rapid-mlx extraction2026-09-10
screening qwen3-next-80b-reap-mlx-mixed3698.3%91.1–99.7Passed 59/60 rapid-mlx extraction2026-09-08
synthesis qwen3-next-80b-reap-mlx-mixed36100.0%94.0–100.0Passed 60/60 mlx_lm extraction2026-09-09
synthesis reap-60b81.7%70.1–89.4Passed 49/60 mlx_lm extraction2026-09-09
synthesis reap-60b-2026091083.3%72.0–90.7Passed 50/60 rapid-mlx extraction2026-09-10
toolcall qwen3-next-80b-reap-mlx-mixed3678.3%66.4–86.9Passed 47/60 rapid-mlx extraction2026-09-09
toolcall reap-60b-2026091078.3%66.4–86.9Passed 47/60 rapid-mlx extraction2026-09-10
toolscale reap-60b-2026091095.0%83.5–98.6Passed 38/40 rapid-mlx extraction2026-09-10
worldview reap-60b47.5%32.9–62.5Passed 19/40 mlx_lm extraction2026-09-09
worldview reap-60b-2026091047.5%32.9–62.5Passed 19/40 rapid-mlx extraction2026-09-10

Runs marked unenforced were measured on a suite that requires constrained decoding, on a server shown not to apply the schema: mlx_lm accepts it and ignores it, and a few other servers and models break it. Those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Corrections 2 to the record published 21 October 2026
  • 2 runs voided (the contract run of 8 September 2026; the screening run of 8 September 2026). Every request failed (HTTP 404: the label was sent as the model id), so nothing was measured.

    Corrected 24 October 2026 · a mechanical fix

The record How it was made · how it was served · what went wrong · what happened to it

Rebuilt from the lab's records by Claude, an AI model. The results above come straight from the automated runs. How these pages are made.

Fileqwen3-next-80b-reap-mlx-mixed36, ~28 GB
FormatMLX mixed_3_6 (4-bit base, 542 overrides, 487 at 3-bit), made here
Where fromsame, 2026-09-08
Statusdeleted 2026-09-10 23:58Z
Hugging Facenot on HF

How it was made

  • Input: the Samsung safetensors (Qwen3-Next-80B-A3B-Instruct REAP safetensors source (SamsungSAILMontreal)), first copy.
  • Command: eval/reap_mlx.sh: mlx_lm convert --hf-path reap-src --mlx-path qwen3-next-80b-reap-mlx-mixed36 -q --q-bits 4 --q-group-size 64 --quant-predicate mixed_3_6, mlx-venv (older mlx). Done by 2026-09-08 20:06Z.
  • Size: ~28 GB (28.1 GB). Two shard sizes were recorded, 5,221,725,661 and 5,353,985,969 bytes; the full shard list and total bytes are not on record.
  • What the header hides: it declares bits: 4, but the config carries 542 per-layer overrides: 487 tensors at 3-bit (including embed_tokens) and 55 at 6-bit, so the build is predominantly 3-bit. The counts were read from the config at 23:53:30Z on 2026-09-10, minutes before deletion. The override map itself was never written down.
  • Who asked: the assistant chose mixed_3_6 unprompted as the "more interesting target"; Alan's 19:02Z "So reap mlx is next then" approved the pipeline.

How it was served

labeldates (UTC)binary and versionctx / -np / KV / -fa / --jinja / --reasoning-formatharness profilewhy this label exists
reapmix36-rapid-mlx2026-09-08 20:08:15Z (finish 21:08 BST)rapid-mlx 0.13.4n/aextractionfirst pass; label used as the wire id, 0 measured (HTTP 404)
qwen3-next-80b-reap-mlx-mixed36screening 2026-09-08 20:38:11Z; contract/2 20:50:38Z; toolcall/1 2026-09-08 23:01:59Z; synthesis/1 2026-09-09 16:51:56Zrapid-mlx 0.13.4 serve <path> --no-spec-decode (no parser flags on the recorded lines); synthesis/1 on mlx_lm 0.31.3wire id = model pathextractionhead-to-head against the IQ4_XS and the 4-bit build
reap-60b2026-09-09 17:39Z synthesis/2, 18:26Z worldview/1 (finish 18:39, 19:26 BST)mlx_lm server --model <path>, 0.31.3n/aextractionthe REAP slot in the role-sized fleet
reap-60b-202609102026-09-10 12:56:54Z to 13:06:04Zrapid-mlx 0.13.4: serve <path> --served-model-name <path> --enable-auto-tool-choice --tool-call-parser qwen3 --reasoning-parser qwen3n/aextractionthe 2026-09-10 "miss REAP" re-bench on the day's exact configuration
(oMLX pool alias)2026-09-09 09:18Z to 12:20ZoMLX 0.6.4n/an/aconsolidation experiment; the toolcall row it produced sits under the IQ4_XS label and is voided

Seats held (all informal): an on-demand accuracy seat in the 2026-09-09 plan; the REAP slot of the role-sized fleet from 2026-09-09 16:45Z until the Metal OOM at 18:43Z; the swap-seat occupant for Hermes/dsh trials 2026-09-09 22:04Z to 2026-09-10 ~15:15Z; the seat behind labs-control.sh reap.

Decisions

  • Why made: GLM mixed_3_6 beat plain 4-bit; expected to be "the smallest and fastest thing on your board holding the extraction seat".
  • Why kept after the 2026-09-09 sweep: the 28 GB build was the surviving MLX REAP and the on-demand accuracy seat.
  • What it taught: it was first read as a REAP success on tool use (78.3) and later as a failure on abstention (47.2); the 2026-09-10/11 reconversion showed the override map moved nullschema +37.5 and toolcall -53 in opposite directions.
  • Why deleted: Alan, 2026-09-10 23:54:33Z, "I don’t need mixed, just the 4 bit quant for mlx and high is what I thought we’d done"; deleted 23:58:19Z after repointing hermes-use.sh, labs-control.sh and the dsh settings to the new 4-bit build.

What went wrong

  • Wrong wire id (404s) on the first pass; re-run with the path as id.
  • oMLX contamination. Served by oMLX it scored toolcall 21/60 = 35.0 against 47/60 on rapid-mlx; availability 12/12 became 0/12. Row voided.
  • Parser false zero. Via dsh and rapid-mlx without --tool-call-parser qwen3 --reasoning-parser qwen3 it returned tool calls as literal text; REAP looked tool-blind for four turns on 2026-09-09 evening. The toolcall/1 suite scored 78.3 on 2026-09-08 with no recorded parser flags and again, exactly 47/60, with them on 2026-09-10, so the suite is insensitive to the flag even though the harnesses were not.
  • Metal OOM. 2026-09-09 18:43Z: kIOGPUCommandBufferCallbackError, 18.4 GB swap; the process kept answering /v1/models while unable to infer, a /v1/models health check cannot tell loaded from working.
  • The 2026-09-10 mix-up (family doc): this build's 47.2 nullschema was first blamed on rapid-mlx not enforcing the schema, which the sentinel probe disproved, then compared with a 95.8 that belongs to a different model.
  • Override map lost. Deleted before the map was written down: its scores are on record, its recipe is not.
  • Standing checks. Partial runs: none (rows match). Stale build stamps: none flagged; rapid-mlx 0.13.4 and mlx_lm 0.31.3 are stamped. Missing flags: the 2026-09-08 rapid-mlx lines have no tool or reasoning parser.

What happened to it

Converted 2026-09-08 ~20:00Z → benched 20:07Z → oMLX pool 2026-09-09 → the fleet → rapid-mlx swap seat → deleted 2026-09-10 23:58:19Z (28.1 GB). No RESTORE entry for the build (RESTORE-deleted-20260911.sh mentions it as unrecoverable). Re-creatable only in principle: source plus --quant-predicate mixed_3_6, but mlx_lm's predicate output was not recorded and the override map cannot be matched.