Working drawing 02 · Cloud← Benchmarks
Cloud candidates.
Three models served from Ollama's cloud, put through the same locked suites as the local models. They ran on Ollama's hardware, not ours: chat stays in the cloud, and this is how the candidates for it measured.
- Models
- 3
- Suites
- 13
- Run
- 2026-10-17
Ollama cloud (remote inference); harness on Mac Studio M1 Max
Ollama 0.12.6, OpenAI-compatible /v1/chat/completions, cloud models
Profile: extraction (temperature 0, max_tokens 2048, thinking=False requested)
Results3 models · 13 suites · 95% intervals
Each cell is the accuracy, passed of measured, and the 95% interval. Same suites, datasets and graders as the local runs; read the caveats before comparing them.
| Suite | gpt-oss:120b | gemma4:31b | mistral-large-3:675b |
|---|---|---|---|
| tagger | 87.5%210/240 · 82.7–91.1 | 97.1%233/240 · 94.1–98.6 | 87.9%211/240 · 83.2–91.5 |
| tagger_nullable | 96.7%232/240 · 93.6–98.3 | 69.6%167/240 · 63.5–75.1 | 97.9%235/240 · 95.2–99.1 |
| needle | 64.5%71/110 · 55.3–72.9 | 62.7%69/110 · 53.4–71.2 | 61.8%68/110 · 52.5–70.4 |
| enrichment | 81.5%44/54 · 69.2–89.6 | not scored | 75.9%41/54 · 63.1–85.4 |
| epistemics | 97.9%47/48 · 89.1–99.6 | 100.0%48/48 · 92.6–100.0 | 100.0%48/48 · 92.6–100.0 |
| synthesis | 55.0%33/60 · 42.5–66.9 | 91.7%55/60 · 81.9–96.4 | 58.3%35/60 · 45.7–69.9 |
| worldview | 52.5%21/40 · 37.5–67.1 | 100.0%40/40 · 91.2–100.0 | 77.5%31/40 · 62.5–87.7 |
| toolcall | 63.3%38/60 · 50.7–74.4 | 33.3%20/60 · 22.7–45.9 | 63.3%38/60 · 50.7–74.4 |
| toolscale | 87.5%35/40 · 73.9–94.5 | 100.0%40/40 · 91.2–100.0 | 97.5%39/40 · 87.1–99.6 |
| tools | 86.7%13/15 · 62.1–96.3 | 93.3%14/15 · 70.2–98.8 | 100.0%15/15 · 79.6–100.0 |
| screening | 91.7%55/60 · 81.9–96.4 | 91.7%55/60 · 81.9–96.4 | 98.3%59/60 · 91.1–99.7 |
| contract | 50.0%24/48 · 36.4–63.6 | 25.0%12/48 · 14.9–38.8 | 58.3%28/48 · 44.3–71.2 |
| chainwalk | 0.0%0/2 · 0.0–65.8 | 50.0%1/2 · 9.5–90.5 | 0.0%0/2 · 0.0–65.8 |
CostTokens and speed, on Ollama's hardware
| Model | Tokens in | Tokens out | Seconds a case, tagger (median) |
|---|---|---|---|
| gpt-oss:120b | 2,655,051 | 343,097 | 1.92 |
| gemma4:31b | 2,917,818 | 70,867 | 0.66 |
| mistral-large-3:675b | 2,823,934 | 80,113 | 2.23 |
Caveats8, as recorded
- thinking=False was NOT applied: Ollama /v1 drops the llama.cpp thinking-off fields the harness sent. gpt-oss:120b-cloud reasoned on every request (~470 output tokens/case), so its numbers are thinking-ON. gemma4:31b-cloud and mistral-large-3:675b-cloud do not think by default on Ollama, so theirs are thinking-off as intended.
- grounding and nullschema not run: Ollama ignores response_format json_schema (honours_schema() = False), and the runner refuses them by design.
- shapegen (DatasetDrift, SUITE_VERSION not bumped) and sitebuild (KeyError 'nesting') fail in the harness for every model; not scored.
- gemma4:31b-cloud enrichment crashed the grader on a JSON-array reply at case 24; not scored.
- No /props on Ollama: server_runtime is empty, so every Ollama run shares a configuration_id shape; not one-to-one comparable with llama.cpp seat runs.
- synthesis/2 string matching misses Unicode punctuation gpt-oss emits (U+2011, U+202F, fullwidth brackets); an offline re-grade with those normalised scores gpt-oss 43/60 vs 33/60 as graded. The graded number is what is recorded here.
- chainwalk has 2 cases; treat as uninformative.
- Speeds include Ollama cloud's own hardware and load, which were not controlled.
Not scored3 models
- deepseek-v4.1-flash:cloud: abandoned after 39 cases: thinks 1.4-1.9k tokens/case and ran to the full 8192 cap on 7/39 even under the synthesis profile
- glm-5.3-flash:cloud: abandoned after 101 cases: thinking-on run judged the wrong configuration for agentic suites
- beam:cloud: 6-case trial only: always thinks; no off switch on Ollama
Follow-up: Harness PR #5 (native Ollama connector + thinking-off precondition on every backend).