FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Model← Models

Gemma 4 E2B

2B dense, q4_0, 5.0 GB.

Parameters
2B
Quant
q4_0
Size
5.0 GB
Family
gemma4
RecordStatus · runtime · seat

gemma4-e2b

Measured, not seateddenseresidentllama.cppmlx

Model card ↗

Outcome as of 20 September 2026, from the seat registry. The write-up below was made when the model was tested.

Why it was tested

The base model for the tagger fine-tune, and the smallest thing worth measuring.

What came of it

85.8% on tagger. Notable mainly for beating its own fine-tuned descendant by 11.6 points.

Derived from this: tagger-e2b-v1 (LoRA fine-tune).

Recorded runs9 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
chainwalk0.0%0.0–65.8Passed 0/2llama.cppagentic-fast2026-08-24
enrichment75.9%63.1–85.4Passed 41/54llama.cppstudy-nothink2026-08-24
epistemics68.8%54.7–80.1Passed 33/48llama.cppstudy-nothink2026-08-24
grounding61.6%52.8–69.7Passed 77/125llama.cppextraction2026-08-24
nullschema12.5%6.7–22.1Passed 9/72llama.cppextraction2026-08-24
screening91.7%81.9–96.4Passed 55/60llama.cppextraction2026-08-24
tagger91.2%87.0–94.2Passed 219/240llama.cppstudy-nothink2026-08-23
tools100.0%79.6–100.0Passed 15/15llama.cppagentic-fast2026-08-24
toolscale100.0%91.2–100.0Passed 40/40llama.cppagentic-fast2026-08-24

Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Every run log entry for this model (10)
FinishedSuiteRuntime · profileScoreCoverageGrader · dataset · config
toolcalltoolcall/1not recordedextraction0.0%Coverage 60/60g 804e8fc9457649d8d 10af88c170513155c 022b5819dd82ff96
screeningscreening/1llama.cppextraction91.7%Coverage 60/60g 0661b49a10be2d2ad 64c97ca2269c29afc 905fc96bd8c3e874
nullschemanullschema/1llama.cppextraction12.5%Coverage 72/72g d7448e0f5fc3f9afd 3b91280bb40372efc e268490afdec64ce
groundinggrounding/1llama.cppextraction61.6%Coverage 125/125g 30b933dcc6a7e2d4d dcb396faf8b83b0ac dee67653e2de45c6
toolscaletoolscale/1llama.cppagentic-fast100.0%Coverage 40/40g 1bae1f3e1b606eaed b24abb108c409178c 3860ab39a73cff08
enrichmentenrichment/3llama.cppstudy-nothink75.9%Coverage 54/54g 7d661b93852b5ffdd 75361864baa727abc 94079faf08beb84b
epistemicsepistemics/4llama.cppstudy-nothink68.8%Coverage 48/48g 2a2f7e4e90d6dd63d 4bce453fd376553cc e5abf1e2bbe8c2c8
chainwalkchainwalk/2llama.cppagentic-fast0.0%Coverage 2/2g 48abd22e0117fddad 129a9e74408c8919c eef56b755300de59
toolstools/1llama.cppagentic-fast100.0%Coverage 15/15g d6a32c09e57bea7bd 33a44812f0ad4c39c 33f8efb324ad39aa
taggertagger/2llama.cppstudy-nothink91.2%Coverage 240/240g 28983856e4bf2dd9d 7ef587bf33e5788ac 72d758d807701d51
FamilyOthers tested from the gemma4 family
  • Gemma 4 12BSeated · Tag

    12B · dense · q4_0 QAT · 7.0 GB

    Tested as a cheap tagger after the 80B proved expensive for a task that runs constantly.

  • tagger-e2b-v1 (LoRA fine-tune)Rejected

    2B · dense · MLX 4-bit · 3.0 GB

    A custom tagger, fine-tuned with MLX LoRA on 270 turns, intended to replace the general model in the tagging seat.