FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Model← Models

Nemotron 3.5 Lightning 30B

30B mixture of experts, 3B active, Q4_K_M, 24.0 GB.

Parameters
30B / 3B active
Quant
Q4_K_M
Size
24.0 GB
Family
nemotron
RecordStatus · runtime · seat

nemotron-lightning-30b

RemovedMoEdeletedllama.cppmlx

Model card ↗

Outcome as of 20 September 2026, from the seat registry. The write-up below was made when the model was tested.

Why it was tested

Tested as a fast MoE, 45.9 tok/s, and briefly the best raw scorer on some suites.

What came of it

Deleted. 15.3% nullschema through llama.cpp and 9.7% through MLX, the worst abstention measured. Fast and confident is the wrong combination for extraction.

Recorded runs3 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
grounding60.0%51.2–68.2Passed 75/125llama.cppextraction2026-09-05
nullschema15.3%8.8–25.3Passed 11/72llama.cppextraction2026-09-05
screening93.3%84.1–97.4Passed 56/60llama.cppextraction2026-09-05

Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Every run log entry for this model (3)
FinishedSuiteRuntime · profileScoreCoverageGrader · dataset · config
nullschemanullschema/1llama.cppextraction15.3%Coverage 72/72g d7448e0f5fc3f9afd 3b91280bb40372efc 8f3225e9b30dcc6b
screeningscreening/1llama.cppextraction93.3%Coverage 60/60g 0661b49a10be2d2ad 64c97ca2269c29afc bd58c270aedc9f3d
groundinggrounding/1llama.cppextraction60.0%Coverage 125/125g 30b933dcc6a7e2d4d dcb396faf8b83b0ac f734b41b3caabfdd