FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Model← Models

tagger-e2b-v1 (LoRA fine-tune)

2B dense, MLX 4-bit, 3.0 GB.

Parameters
2B
Quant
MLX 4-bit
Size
3.0 GB
Family
gemma4
RecordStatus · runtime · seat

tagger-e2b-v1

Rejecteddenseresidentmlx

Outcome as of 20 September 2026, from the seat registry. The write-up below was made when the model was tested.

Why it was tested

A custom tagger, fine-tuned with MLX LoRA on 270 turns, intended to replace the general model in the tagging seat.

What came of it

Cutover cancelled. Scored 74.2% against its own base model's 85.8% on the same runtime, profile and 240 pairs, with non-overlapping intervals. The fine-tune cost 11.6 points and bought nothing. The training data needs revisiting, not the runtime.

Fine-tuned from Gemma 4 E2B.

Recorded runs1 scored · 95% intervals
SuiteScore95% interval PassedRuntime · profileRecorded
tagger77.1%71.4–81.9Passed 185/240llama.cppextraction2026-08-23

Runs marked unenforced were measured through mlx_lm on a suite that requires constrained decoding. mlx_lm accepts the schema and ignores it, so those numbers do not measure what the suite is designed to measure. They are kept and labelled rather than deleted.

Every run log entry for this model (1)
FinishedSuiteRuntime · profileScoreCoverageGrader · dataset · config
taggerllama.cppextraction77.1%Coverage 240/240g —d —c —
FamilyOthers tested from the gemma4 family
  • Gemma 4 12BSeated · Tag

    12B · dense · q4_0 QAT · 7.0 GB

    Tested as a cheap tagger after the 80B proved expensive for a task that runs constantly.

  • Gemma 4 E2BMeasured, not seated

    2B · dense · q4_0 · 5.0 GB

    The base model for the tagger fine-tune, and the smallest thing worth measuring.