FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Model← Models

Qwen3-Next-80B-A3B Instruct, REAP pruned

60B mixture of experts, ~3B active, IQ4_XS, 30.4 GB.

Parameters
60B / ~3B active
Quant
IQ4_XS
Size
30.4 GB
RecordStatus · runtime · seat

qwen3-next-80b-reap

Measured, not seatedMoEresidentllama.cpp

Model card ↗

Outcome as of 20 September 2026, from the seat registry. The write-up below was made when the model was tested.

Why it was tested

Tested to find out whether expert pruning costs the extraction seat its abstention. The full model holds that seat at 40 GB, which is most of the memory budget on a 64 GB machine.

What came of it

It costs nothing measurable. nullschema 97.2 percent, the same 70 of 72 as the unpruned model with the same two failures. grounding within noise at 87.2 against 88.8, screening better at 98.3 against 91.7, and 41.4 tok/s against 37.7. Ten gigabytes smaller and slightly faster.

Recorded runs4 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
contract0.0%0.0–7.4Passed 0/48llama.cppextraction—
grounding87.2%80.2–92.0Passed 109/125llama.cppextraction2026-09-08
nullschema97.2%90.4–99.2Passed 70/72llama.cppextraction2026-09-08
screening98.3%91.1–99.7Passed 59/60llama.cppextraction2026-09-08

Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Every run log entry for this model (5)
FinishedSuiteRuntime · profileScoreCoverageGrader · dataset · config
toolcalltoolcall/1not recordedextraction35.0%Coverage 60/60g 804e8fc9457649d8d 10af88c170513155c e9925436d57909a1
contractcontract/2not recordedextractionn/aCoverage 0% coverageg 486601ba23034dd6d 1fe53d30cb5480f6c cbb517440b51fc70
nullschemanullschema/1not recordedextraction97.2%Coverage 72/72g d7448e0f5fc3f9afd 3b91280bb40372efc cc55008d40e60573
groundinggrounding/1not recordedextraction87.2%Coverage 125/125g 30b933dcc6a7e2d4d dcb396faf8b83b0ac 7e1bbf58551d87ef
screeningscreening/1not recordedextraction98.3%Coverage 60/60g 0661b49a10be2d2ad 64c97ca2269c29afc 81efb9eaae394534
FamilyOthers tested from the qwen3-next family
  • Qwen3-Next-80B-A3B InstructRemoved

    80B · MoE · IQ4_XS · 40.0 GB

    The incumbent. Chosen because 3B active parameters give MoE speed at 80B quality, and it fits at 40GB with room for a long context.

  • Qwen3-Next-80B-A3B ThinkingRejected

    80B · MoE · IQ4_XS · 40.0 GB

    Tested to see whether reasoning improves extraction, on the assumption that thinking helps.