Suite · tagger← Suites
Tagger contract compliance
Does the model produce tags that satisfy the contract?
- Cases
- 480
- Scored runs
- 29
- Models
- 29
- Top score
- 93.8%
SpecificationMethod · grading · caveat
- Method
- 240 conversation turn pairs are tagged for category, topics and entities. The category must come from a fixed list, topics are bounded in number, and every entity must appear in the turn.
- Grading
- Rule based. Category outside the list, topic count out of range, or an entity absent from the source all fail the case.
- Caveat
- Contract compliance is not the same as tag quality. A human check of the gold pairs is still outstanding.
- Version
tagger/2- Cases
- 480
Recorded runs29 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite gemma4-12b-ab 93.8% 95% interval 89.9–96.2
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| gemma4-12b-ab | 93.8% | 89.9–96.2 | Passed 225/240 | llama.cppextraction | 2026-08-24 |
| gemma4-12b-b10618 | 93.8% | 89.9–96.2overlaps the top interval | Passed 225/240 | llama.cppextraction | 2026-08-31 |
| qwen3-next-80b-tag | 93.3% | 89.4–95.9overlaps the top interval | Passed 224/240 | llama.cppextraction | 2026-08-31 |
| Gemma 4 E2Bgemma4-e2b | 91.2% | 87.0–94.2overlaps the top interval | Passed 219/240 | llama.cppstudy-nothink | 2026-08-23 |
| gemma4-e2b-ab | 91.2% | 87.0–94.2overlaps the top interval | Passed 219/240 | llama.cppextraction | 2026-08-24 |
| qwen3.6-35b-tag | 90.8% | 86.5–93.9overlaps the top interval | Passed 218/240 | llama.cppextraction | 2026-08-31 |
| edge0-35b | 88.3% | 83.7–91.8overlaps the top interval | Passed 212/240 | llama.cppextraction | 2026-09-16 |
| bonsai-27b-mlx2-m4pro | 87.9% | 83.2–91.5overlaps the top interval | Passed 211/240 | mlxextraction | 2026-09-14 |
| bonsai-27b-mlx2-cortext | 87.5% | 82.7–91.1overlaps the top interval | Passed 210/240 | mlxextraction | 2026-09-15 |
| BTL-4 (Q4_K_M)btl-4-q4km | 87.1% | 82.2–90.7overlaps the top interval | Passed 209/240 | llama.cppextraction | 2026-09-15 |
| gpt-oss-20b | 86.2% | 81.3–90.0overlaps the top interval | Passed 207/240 | llama.cppextraction | 2026-09-12 |
| gemma4-e2b-mlx-4bit | 85.8% | 80.9–89.7 | Passed 206/240 | mlxextraction | 2026-09-07 |
| ornith-35b-tagger | 85.8% | 80.9–89.7 | Passed 206/240 | llama.cppextraction | 2026-09-20 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 85.8% | 80.9–89.7 | Passed 206/240 | llama.cppextraction | 2026-09-14 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 85.4% | 80.4–89.3 | Passed 205/240 | llama.cppextraction | 2026-09-15 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 85.0% | 79.9–89.0 | Passed 204/240 | llama.cppextraction | 2026-09-15 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 83.3% | 78.1–87.5 | Passed 200/240 | llama.cppextraction | 2026-09-14 |
| tagger-e2b-v1 (LoRA fine-tune)tagger-e2b-v1 | 77.1% | 71.4–81.9 | Passed 185/240 | llama.cppextraction | 2026-08-23 |
| minicpm5-2b-q8-np1 | 76.7% | 70.9–81.6 | Passed 184/240 | llama.cppextraction | 2026-09-17 |
| minicpm5-2b-exl3 | 76.2% | 70.5–81.2 | Passed 183/240 | llama.cppextraction | 2026-09-17 |
| minicpm5-2b-q8-nojinja | 76.2% | 70.5–81.2 | Passed 183/240 | llama.cppextraction | — |
| minicpm5-2b-q8-np4-today | 76.2% | 70.5–81.2 | Passed 183/240 | llama.cppextraction | — |
| Gemma 4 12Bgemma4-12b | 75.0% | 69.2–80.1 | Passed 180/240 | llama.cppstudy-nothink | — |
| tagger-e2b-v1-dl-tagger-e2b-v1 | 74.2% | 68.3–79.3 | Passed 178/240 | mlxextraction | 2026-09-07 |
| minicpm5-2b | 51.7% | 45.4–57.9 | Passed 124/240 | llama.cppextraction | 2026-09-15 |
| gemma4-e4b | 45.4% | 41.0–49.9 | Passed 218/480 | llama.cppstudy-nothink | — |
| ternary-bonsai-8b | 43.8% | 37.6–50.1 | Passed 105/240 | llama.cppextraction | 2026-09-17 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 28.3% | 23.0–34.3 | Passed 68/240 | llama.cppextraction | 2026-09-15 |
| qwen3.8-iq1s | 14.0% | 9.0–21.0 | Passed 18/129 | llama.cppstudy-nothink | — |
Run log31 logged runs of this suite
Every logged run, newest first (31)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| ornith-35b-tagger | not recordedextraction | 85.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 02e7b350ee762892 | |
| minicpm5-2b-q8-nojinja | not recordedextraction | 76.9% | Coverage 99.2% coverage | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 0153d351e0ec30ee | |
| minicpm5-2b-q8-np4-today | not recordedextraction | 76.9% | Coverage 99.2% coverage | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 0d285fb9372dab00 | |
| minicpm5-2b-q8-np1 | not recordedextraction | 76.7% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac c1fd37845b5547c8 | |
| minicpm5-2b-exl3 | not recordedextraction | 76.2% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 58ec2a66d452536c | |
| ternary-bonsai-8b | not recordedextraction | 43.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 1b7169aea80ec684 | |
| ternary-bonsai-8b | not recordedextraction | 36.4% | Coverage 11/11 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 1b7169aea80ec684 | |
| edge0-35b | not recordedextraction | 88.3% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 7149c1e662a034c9 | |
| edge0-35b | not recordedextraction | 85.5% | Coverage 83/83 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 7149c1e662a034c9 | |
| minicpm5-2b | not recordedextraction | 51.7% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 8bc8889aa4bec4ee | |
| btl-4-q4km | not recordedextraction | 87.1% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 10dcbcd1b3bb7b91 | |
| btl-4 | not recordedextraction | 85.0% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 81d3e449bf19eaea | |
| bonsai-27b-1bit | not recordedextraction | 85.4% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac d75dadae6c07b420 | |
| bonsai-27b-mlx2-cortext | mlxextraction | 87.5% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac a7bdc6604033d727 | |
| bonsai-8b-1bit | not recordedextraction | 28.3% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac e931e287089b3229 | |
| bonsai-27b-mlx2-m4pro | mlxextraction | 87.9% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 821ca1ce68849b25 | |
| ternary-bonsai-27b | not recordedextraction | 85.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 773ed13c720ca034 | |
| qwen3.8-9b-distill | not recordedextraction | 83.3% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac d2642a91c9a03367 | |
| gpt-oss-20b | not recordedextraction | 86.2% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac b9bd7f33427388fd | |
| tagger-e2b-v1-dl-tagger-e2b-v1 | llama.cppextraction | 74.2% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 26b77eacaf1fc207 | |
| gemma4-e2b-mlx-4bit | mlxextraction | 85.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 1623b2cdd0ccbfaf | |
| qwen3-next-80b-tag | llama.cppextraction | 93.3% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac bb9d18596364b9f4 | |
| qwen3.6-35b-tag | llama.cppextraction | 90.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 336518ff1b1d0973 | |
| gemma4-12b-b10618 | llama.cppextraction | 93.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac c947e8a080eadfe2 | |
| gemma4-12b-ab | llama.cppextraction | 93.8% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 3c8ef5d04a11b585 | |
| gemma4-e2b-ab | llama.cppextraction | 91.2% | Coverage 240/240 | g 63e04d139a0e5ed3d 7ef587bf33e5788ac 4619919ac18affd5 | |
| qwen3.8-iq1s | llama.cppstudy-nothink | 31.6% | Coverage 44.2% coverage | g 956e03bbffa79aa4d 7ef587bf33e5788ac f6700e17f70a9c2a | |
| gemma4-12b | llama.cppstudy-nothink | 93.2% | Coverage 80.3% coverage | g 28983856e4bf2dd9d 7ef587bf33e5788ac d10e58be49cb1d55 | |
| gemma4-e4b | llama.cppstudy-nothink | 90.8% | Coverage 50% coverage | g 28983856e4bf2dd9d 7ef587bf33e5788ac 60e6c55236269057 | |
| gemma4-e2b | llama.cppstudy-nothink | 91.2% | Coverage 240/240 | g 28983856e4bf2dd9d 7ef587bf33e5788ac 72d758d807701d51 | |
| tagger-e2b-v1 | llama.cppextraction | 77.1% | Coverage 240/240 | g —d —c — |
Findings6 that name this suite
- 2026-09-20Typed decisions split across a routing table, not a winner; the only arm you can put a threshold on matters more than the most accurate one
- 2026-09-17EXL3/QTIP at 4 bpw costs nothing measurable in capability and three quarters of the throughput
- 2026-09-17One baseline was 25 points off and nothing in the stamp could explain it; the other six reproduced exactly
- 2026-09-15MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites
- 2026-09-10The 6.7 GB tagger is the most epistemically reliable model in the fleet
- 2026-09-07The tagger fine-tune scored below its own base model