Suite · epistemics← Suites
Given versus prior knowledge
Does the model answer from the document it was given, or from what it already knows?
- Cases
- 48
- Scored runs
- 25
- Models
- 25
- Top score
- 100.0%
SpecificationMethod · grading · caveat
- Method
- Documents contain facts that contradict widely known information. The model is asked questions whose answers differ depending on which source it trusts.
- Grading
- A case passes when the answer matches the supplied document rather than general knowledge.
- Caveat
- Several models now score 100 on this suite, so it has stopped discriminating.
- Version
epistemics/4- Cases
- 48
Recorded runs25 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite Agents-A1 35B 100.0% 95% interval 92.6–100.0
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| Agents-A1 35Bagents-a1-35b | 100.0% | 92.6–100.0 | Passed 48/48 | llama.cppstudy-nothink | 2026-08-24 |
| bonsai2-27b-pq2 | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppstudy-nothink | 2026-09-18 |
| bonsai2-27b-ptq1 | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppstudy-nothink | 2026-09-18 |
| BTL-4 Compact (IQ2_XXS)btl-4 | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppstudy-nothink | 2026-09-15 |
| BTL-4 (Q4_K_M)btl-4-q4km | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppstudy-nothink | 2026-09-15 |
| Gemma 4 12Bgemma4-12b | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppstudy-nothink | 2026-08-24 |
| ornith-1.5-9b | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppextraction | 2026-09-12 |
| Qwen3.8 27Bqwen3.8-27b | 100.0% | 92.6–100.0overlaps the top interval | Passed 48/48 | llama.cppstudy-nothink | 2026-08-24 |
| edge0-35b | 97.9% | 89.1–99.6overlaps the top interval | Passed 47/48 | llama.cppstudy-nothink | 2026-09-16 |
| gemma4-e4b | 97.9% | 89.1–99.6overlaps the top interval | Passed 47/48 | llama.cppstudy-nothink | 2026-08-24 |
| bonsai-27b-mlx2-cortext | 89.6% | 77.8–95.5overlaps the top interval | Passed 43/48 | mlxstudy-nothink | 2026-09-14 |
| bonsai-27b-mlx2-m4pro | 89.6% | 77.8–95.5overlaps the top interval | Passed 43/48 | mlxstudy-nothink | 2026-09-14 |
| Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill | 89.6% | 77.8–95.5overlaps the top interval | Passed 43/48 | llama.cppstudy-nothink | 2026-09-14 |
| Bonsai-27B (Q1_0)bonsai-27b-1bit | 87.5% | 75.3–94.1overlaps the top interval | Passed 42/48 | llama.cppstudy-nothink | 2026-09-14 |
| qwen3.5-4b-base | 85.4% | 72.8–92.8overlaps the top interval | Passed 41/48 | llama.cppstudy-nothink | 2026-09-13 |
| Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b | 85.4% | 72.8–92.8overlaps the top interval | Passed 41/48 | llama.cppstudy-nothink | 2026-09-14 |
| minicpm5-2b-exl3 | 81.2% | 68.1–89.8 | Passed 39/48 | llama.cppstudy-nothink | 2026-09-17 |
| Bonsai-8B (Q1_0)bonsai-8b-1bit | 79.2% | 65.7–88.3 | Passed 38/48 | llama.cppstudy-nothink | 2026-09-14 |
| minicpm5-2b-q8-np1 | 79.2% | 65.7–88.3 | Passed 38/48 | llama.cppstudy-nothink | 2026-09-17 |
| minicpm5-2b | 77.1% | 63.5–86.7 | Passed 37/48 | llama.cppstudy-nothink | 2026-09-15 |
| glm-4.7-flash-q40 | 75.0% | 61.2–85.1 | Passed 36/48 | llama.cppextraction | 2026-09-12 |
| neohorse-1-4b | 75.0% | 61.2–85.1 | Passed 36/48 | llama.cppstudy-nothink | 2026-09-13 |
| ternary-bonsai-8b | 72.9% | 59.0–83.4 | Passed 35/48 | llama.cppstudy-nothink | 2026-09-17 |
| Gemma 4 E2Bgemma4-e2b | 68.8% | 54.7–80.1 | Passed 33/48 | llama.cppstudy-nothink | 2026-08-24 |
| qwen3.8-iq1s | 56.2% | 42.3–69.3 | Passed 27/48 | llama.cppstudy-nothink | — |
Run log25 logged runs of this suite
Every logged run, newest first (25)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| bonsai2-27b-ptq1 | not recordedstudy-nothink | 100.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 9255c41b05f7da62 | |
| bonsai2-27b-pq2 | not recordedstudy-nothink | 100.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 3e65308eee2e3c69 | |
| minicpm5-2b-q8-np1 | not recordedstudy-nothink | 79.2% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc d8638e6e2d8d038b | |
| minicpm5-2b-exl3 | not recordedstudy-nothink | 81.2% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 18bb944eee207790 | |
| ternary-bonsai-8b | not recordedstudy-nothink | 72.9% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 22502c57cf79b73d | |
| edge0-35b | not recordedstudy-nothink | 97.9% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 63bebcfa1d0c9c54 | |
| minicpm5-2b | not recordedstudy-nothink | 77.1% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 8bf3c6f0ca1fa01a | |
| btl-4-q4km | not recordedstudy-nothink | 100.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 0689f579cb5fa875 | |
| btl-4 | not recordedstudy-nothink | 100.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 438aa2f62c6153af | |
| bonsai-27b-1bit | not recordedstudy-nothink | 87.5% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 2f1d2e8b3a422ec6 | |
| bonsai-8b-1bit | not recordedstudy-nothink | 79.2% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc f1ce82d657bdf4e2 | |
| bonsai-27b-mlx2-cortext | mlxstudy-nothink | 89.6% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc f8818645d3ba1522 | |
| bonsai-27b-mlx2-m4pro | mlxstudy-nothink | 89.6% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 3c03af0d96b59518 | |
| ternary-bonsai-27b | not recordedstudy-nothink | 85.4% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 321e00a36a5faaeb | |
| qwen3.8-9b-distill | not recordedstudy-nothink | 89.6% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc a0e9adc5c6542d53 | |
| qwen3.5-4b-base | not recordedstudy-nothink | 85.4% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 211ae40d03020e0e | |
| neohorse-1-4b | not recordedstudy-nothink | 75.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc a4f7cdc18f2e769e | |
| ornith-1.5-9b | not recordedextraction | 100.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 54cc298fd20d22af | |
| glm-4.7-flash-q40 | not recordedextraction | 75.0% | Coverage 48/48 | g acf84972a0ad5f38d 4bce453fd376553cc 0550121d973797df | |
| qwen3.8-iq1s | llama.cppstudy-nothink | 77.1% | Coverage 72.9% coverage | g 2a2f7e4e90d6dd63d 4bce453fd376553cc 4bfecdedff811923 | |
| qwen3.8-27b | llama.cppstudy-nothink | 100.0% | Coverage 48/48 | g 2a2f7e4e90d6dd63d 4bce453fd376553cc 541a385213b500cb | |
| agents-a1-35b | llama.cppstudy-nothink | 100.0% | Coverage 48/48 | g 2a2f7e4e90d6dd63d 4bce453fd376553cc 2101ffcef80989dc | |
| gemma4-12b | llama.cppstudy-nothink | 100.0% | Coverage 48/48 | g 2a2f7e4e90d6dd63d 4bce453fd376553cc b6ac24eba6a83a5d | |
| gemma4-e4b | llama.cppstudy-nothink | 97.9% | Coverage 48/48 | g 2a2f7e4e90d6dd63d 4bce453fd376553cc 37147fa352af2d7a | |
| gemma4-e2b | llama.cppstudy-nothink | 68.8% | Coverage 48/48 | g 2a2f7e4e90d6dd63d 4bce453fd376553cc e5abf1e2bbe8c2c8 |
Findings6 that name this suite
- 2026-09-18A ternary-trained Qwen3.8-27B holds parity on free-form work at 55% of the size, and loses everything it loses on schema-constrained output
- 2026-09-18PTQ1_0 and PQ2_0 produce byte-identical output on a ternary-trained model, so the smaller file is free
- 2026-09-17EXL3/QTIP at 4 bpw costs nothing measurable in capability and three quarters of the throughput
- 2026-09-15IQ2_XXS took 27 points off world knowledge and 17 off tool invocation on identical weights
- 2026-09-15MLX and GGUF score the same on a dense model: mean delta 0.55 points over twelve suites
- 2026-09-13Long-context recall does not degrade with depth; misattribution does the damage, and a string comparison catches all of it