Suite · enrichment← Suites
Memory layer enrichment
Can a local model do the enrichment work the memory layer needs?
- Cases
- 54
- Scored runs
- 17
- Models
- 17
- Top score
- 87.0%
SpecificationMethod · grading · caveat
- Method
- Turns are enriched with structured context. The suite measures whether the enrichment is usable downstream rather than whether it reads well.
- Grading
- Field level checks against the expected structure.
- Caveat
- Small case count. Treat single runs as indicative only.
- Version
enrichment/3- Cases
- 54
Recorded runs17 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite BTL-4 Compact (IQ2_XXS) 87.0% 95% interval 75.6–93.6
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| BTL-4 Compact (IQ2_XXS)btl-4 | 87.0% | 75.6–93.6 | Passed 47/54 | llama.cppextraction | 2026-09-15 |
| BTL-4 (Q4_K_M)btl-4-q4km | 81.5% | 69.2–89.6overlaps the top interval | Passed 44/54 | llama.cppextraction | 2026-09-15 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 79.6% | 67.1–88.2overlaps the top interval | Passed 43/54 | llama.cppextraction | 2026-09-15 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 79.6% | 67.1–88.2overlaps the top interval | Passed 43/54 | llama.cppextraction | 2026-09-14 |
| edge0-35b | 77.8% | 65.1–86.8overlaps the top interval | Passed 42/54 | llama.cppextraction | 2026-09-16 |
| Gemma 4 12Bgemma4-12b | 77.8% | 65.1–86.8overlaps the top interval | Passed 42/54 | llama.cppstudy-nothink | 2026-08-24 |
| Gemma 4 E2Bgemma4-e2b | 75.9% | 63.1–85.4overlaps the top interval | Passed 41/54 | llama.cppstudy-nothink | 2026-08-24 |
| bonsai-27b-mlx2-cortext | 72.2% | 59.1–82.4overlaps the top interval | Passed 39/54 | mlxextraction | 2026-09-15 |
| bonsai-27b-mlx2-m4pro | 72.2% | 59.1–82.4overlaps the top interval | Passed 39/54 | mlxextraction | 2026-09-14 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 72.2% | 59.1–82.4overlaps the top interval | Passed 39/54 | llama.cppextraction | 2026-09-14 |
| gemma4-e4b | 70.4% | 57.2–80.9overlaps the top interval | Passed 38/54 | llama.cppstudy-nothink | 2026-08-24 |
| ternary-bonsai-8b | 70.4% | 57.2–80.9overlaps the top interval | Passed 38/54 | llama.cppextraction | 2026-09-17 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 61.1% | 47.8–73.0 | Passed 33/54 | llama.cppextraction | 2026-09-14 |
| minicpm5-2b | 59.3% | 46.0–71.3 | Passed 32/54 | llama.cppextraction | 2026-09-15 |
| minicpm5-2b-exl3 | 59.3% | 46.0–71.3 | Passed 32/54 | llama.cppextraction | 2026-09-17 |
| minicpm5-2b-q8-np1 | 57.4% | 44.2–69.7 | Passed 31/54 | llama.cppextraction | 2026-09-17 |
| qwen3.8-iq1s | 48.1% | 35.4–61.1 | Passed 26/54 | llama.cppstudy-nothink | — |
Run log17 logged runs of this suite
Every logged run, newest first (17)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| minicpm5-2b-q8-np1 | not recordedextraction | 57.4% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 1fb5d909364039d2 | |
| minicpm5-2b-exl3 | not recordedextraction | 59.3% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 50da51be20c4e83c | |
| ternary-bonsai-8b | not recordedextraction | 70.4% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 2449c0a03ec229c7 | |
| edge0-35b | not recordedextraction | 77.8% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc a0c2477ba575e774 | |
| minicpm5-2b | not recordedextraction | 59.3% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc a602b52d4aa5d258 | |
| btl-4-q4km | not recordedextraction | 81.5% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 3b3b53961682ee7f | |
| btl-4 | not recordedextraction | 87.0% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 0b71856ba217c91c | |
| bonsai-27b-1bit | not recordedextraction | 79.6% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 612035dbfdcf9143 | |
| bonsai-27b-mlx2-cortext | mlxextraction | 72.2% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc c6981047f9ae77d3 | |
| bonsai-8b-1bit | not recordedextraction | 61.1% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc e43f51321b57c883 | |
| bonsai-27b-mlx2-m4pro | mlxextraction | 72.2% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc e41b038c77c413b8 | |
| ternary-bonsai-27b | not recordedextraction | 72.2% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc 075725332e318d6b | |
| qwen3.8-9b-distill | not recordedextraction | 79.6% | Coverage 54/54 | g b51222c2ca38cfdfd 75361864baa727abc d537736901f01a81 | |
| qwen3.8-iq1s | llama.cppstudy-nothink | 60.5% | Coverage 79.6% coverage | g 7d661b93852b5ffdd 75361864baa727abc 11d22e9108889bce | |
| gemma4-12b | llama.cppstudy-nothink | 77.8% | Coverage 54/54 | g 7d661b93852b5ffdd 75361864baa727abc 5367be2a63165de3 | |
| gemma4-e4b | llama.cppstudy-nothink | 70.4% | Coverage 54/54 | g 7d661b93852b5ffdd 75361864baa727abc 60945fac08964a72 | |
| gemma4-e2b | llama.cppstudy-nothink | 75.9% | Coverage 54/54 | g 7d661b93852b5ffdd 75361864baa727abc 94079faf08beb84b |