Model family← Models

Kolibri-1 78B-A3.5B

Aleph Alpha's 78-billion-parameter model, which uses about 3.5 billion for each token. Fast and strong with tools, but weak at judging evidence, and it never took a job here.

Files tested: 3
Scored runs: 7
Smallest Mac: 96 GB for the file we kept

OverviewWhy we tested it · what happened · should you use it

How this page was made. The tests run automatically on our Mac Studio, and every score here comes straight from those runs. This overview was written by Claude, an AI model, from the lab's own records: the run log, notes kept at the time, and the working sessions where each decision was made. Quotes are Alan's words, with spelling fixed. How these pages are made.

Why we tested it

On 4 October Alan asked for a newly released model to benchmark, and picked Kolibri-1 from a shortlist. Aleph Alpha had released it the day before: 78 billion parameters, about 3.5 billion working on each token, built for English and German, with extra training on finding information, declining to answer and calling tools.

Alan: "what can we expect from this, read about the model, apparently it's been trained in areas we care about like grounding".

The question was whether that much model, working at the size of the 3-billion-active models we already run, gives better answers at the same speed. Later came a second question: could one fast, big model take over several of our smaller models' jobs?

What it took

Standard llama.cpp doesn't know Kolibri's design, so every GGUF run needs a community patch; we used Hob-forge's. We started on a community MLX conversion, then made our own Q4_0 from Hob-forge's Q8_0 after Alan said: "4_0 works better". The first attempt failed without printing the word "error", and we lost an hour before adding llama.cpp's --allow-requantize setting.

What happened

The Q4_0 was 47% faster than MLX. It wrote 70.5 tokens a second against 48.1 for the MLX copy.
It was perfect at reaching for and choosing tools: 60 of 60 on our test of when to reach for a tool, and 40 of 40 choosing among many.
It was weaker at judging evidence. On our test of weighing sources it scored 60%, and got none right where a claim rested on one source or had been superseded. Gemma 4 12B, which does that job for us, got 39 of 40. On saying "I don't know" it scored 54.2%.
So it took no job. Alan, weighing it up before the scores came in: "trying to figure out if this is a good or a bad move, the fleet I have just now is pretty versatile". The idea became running it on demand on a second Mac.
Squeezing its conversation memory cost speed. Storing the memory it needs for each token of conversation at 4-bit made it about 10% slower on one conversation and about 20% slower with four at once, and saved almost nothing, because only 10 of its 50 layers keep that memory for the whole conversation.
A fuller test never ran. On 11 October a run with four conversations at once died within a minute on a script error, so six of our extraction tests were never run on it. Our server has a swap seat, one slot where models are swapped in to try, and Alan asked instead: "actually, would a smaller quant fit in the swap seat?" That led us to Sakura, a pruned, smaller version made by someone else.

What it taught us

"MLX is faster" is not a rule. For this model, a Q4_0 on llama.cpp beat MLX by almost half.
Compressing the conversation memory isn't free. On a model where most layers only look back a short way, it saves little and costs speed. Use the default f16.
Being perfect at picking tools doesn't make a model a good judge of evidence.

Should you use it?

If you have room for a 44 GB file, can build llama.cpp with a community patch, and want a fast model for tool calls, it did well here. Leave the conversation memory at f16. Don't rely on it to judge sources or to say "I don't know". For that, Gemma 4 12B did far better here at a fraction of the size.

The files3 tested · each has its own page

Q4_0kept
The one we made and scored, 44.2 GB. Still on disk, not in use.
Made here from the Hob-forge Q8_0, --allow-requantize · On disk · 44.2 GB · 7 scored runs · smallest Mac 96 GB
MLX 4-bit
A community conversion for Apple's MLX, used for our first tests. Deleted the next day to make room for the Q4_0.
Velaia conversion; study and board runs · Deleted 5 Oct 2026 · 44.3 GB · never scored
Q8_0
A near-full-precision community copy, 83.1 GB, downloaded to make the Q4_0 from. Never scored.
Hob-forge two-part GGUF; quantisation source and PPL reference · On disk · 83.1 GB · never scored

The lab's recordEvery decision, with its date · the model's specification

Decisions, in order

2026-10-04 18:36Z, Alan picks Kolibri from a model shortlist. Reasoning: new EU model, claims grounding/abstention training.
2026-10-04 ~18:40Z, runtime choice: MLX 4-bit first (assistant recommendation, accepted via the question prompt; the later summary lists the answers as "MLX 4-bit" among the question-prompt answers). Reasons given: ~1.45x faster on this box per the server runbook, release mlx-lm, no rebuild of the llama.cpp tree that carries the diffusion commit (a lab note). Alternatives: Hob-forge Q4_K_M on patched llama.cpp; Q3_K_S on the other patch (rejected as a clean read on the model).
2026-10-04 19:10Z, Alan: "squish hasn't been used in ages" -> harness client moved to a dedicated eval-venv.
2026-10-05 04:41Z, pre-registered expectation (assistant): the suite is saturated, so the only informative outcome is a miss on CONFLICT; the tech report's fact-check number predicted trouble there. Result: 48/48, every CONFLICT flagged. The fact-check weakness showed up instead on the board's worldview/superseded arm (0/8).
2026-10-05 08:32Z, Alan: "ok, nothing should be running on high to be fair, it's unrealistic" -> full 16-suite board re-run at reasoning_effort: low.
2026-10-05 09:59Z, Alan: "4_0 works better". Assistant, immediately after: "Then we make one.", with Q8_0 -> Q4_0 through the patched llama-quantize as the route. Deletion approvals for the Qwen3-Next-80B GGUF and MLX copies and the Kolibri MLX shards followed at 10:06Z.
2026-10-05 10:09Z, Alan: "trying to figure out if this is a good or a bad move, the fleet I have just now is pretty versatile", then "fair enough, have been eyeing up a m5 pro mac mini 64gb". Seat-board verdict at 12:58Z: consolidation is "a measured no"; role = on-demand generalist on a second host.
2026-10-11 12:33-12:36Z, Alan: "what do I need to do to get kolibri to fit", then "I think we need to gauntlet kolibri properly as n 4 would give concurrency and speed if it can do the job of other seats and simplify overall" and "approved make is so". The window aborted within a minute (see Kolibri-1 Q4_0 (made here from the Hob-forge Q8_0, --allow-requantize), Incidents) and the full-Kolibri gauntlet was never run.
2026-10-11 12:40Z, Alan: "actually, would a smaller quant fit in the swap seat?", then "it may be a way of getting the bigger parameter model in the swap as it's the biggest model and the fastest we have". Outcome: second Q8_0 download, an importance matrix from the Q4_0, and the Sakura family (family-sakura-kolibri-1-365e).
2026-10-11 sweep: a sweep note lists the Kolibri GGUF folder (41 GiB) as "biggest single dir on the box", verdict "second-host role", "Alan's call". Disk state read 2026-10-23: Q4_0 and the second Q8_0 are still present.

What we learned, in the lab's words

GGUF Q4_0 decodes 47% faster than MLX 4-bit on this box: llama-bench tg128 70.52 ± 0.01 t/s, pp512 628.39 against mlx_lm generate 48.1 t/s. The assistant's "MLX is ~1.45x faster" rule of thumb fell over for this model; Alan's remark "k m quants are slower on m1" and "4_0 works better" were what drove the Q4_0 build.
Seat board (seven suites, extraction profile) says it cannot replace the provenance seat: worldview 24/40 with single_source 0/8 and superseded 0/8 against gemma4-12b's 39/40 (table below in Kolibri-1 Q4_0 (made here from the Hob-forge Q8_0, --allow-requantize)). toolscale 40/40 was best on record at the time.
Effort level was the whole difference on the 16-suite board: 12/16 at the template-default high, 13/16 at low. Alan: "ok, nothing should be running on high to be fair, it's unrealistic".
Two corrections after the fact (lab note changelog, 2026-10-09): "fine-grained MoEs do not batch" was an mlx_lm property, not a model property; and the b11396-vs-b11533 batched A/B was invalid because both arms loaded 0.6.0 libraries. The 125.3 t/s four-stream figure was then not reproduced on 2026-10-11: same-day 4-stream totals were ~104 t/s (f16 KV) and ~84 t/s (q4_0 KV) on b11533.
q4_0 KV cache costs Kolibri 7-24% (single-stream 70.75 -> 64.0 t/s; four-stream ~104 -> ~84) for almost no memory saving, because only 10 of 50 layers keep full-length KV. Serve Kolibri-family models with f16 KV.
Same-weights-two-runtimes: MLX 4-bit and llama.cpp Q4_0 are different weights files and different runtimes, so the MLX study numbers (89.4% strict, 624 tasks) and the GGUF seat-board numbers are separate measurements, never one row.

Why it was tested, in the lab's words

Alan, 2026-10-04 18:35Z, opening the session: "I want you to look into a brand new model that's been released so we can benchmark it". He named no model; the assistant listed candidates and Alan picked Kolibri from the question prompt.
Alan, 18:50Z: "what can we expect from this, read about the model, apparently it's been trained in areas we care about like grounding". The assistant read the 189-page tech report and answered with the grounding/abstention training story (Merlin–Arthur protocol) and the caveat that it was trained to trust the document.
Alan, 19:02Z: "am curious as it's the first eu native model aimed at local / sovereignty". The assistant's reply corrected "first" and tied it to the fleet thesis.
By active parameters it sits in the class of the fleet's 3B-active MoEs (qwen3.6-35b-a3b, gemma4-26b-a4b) with 2-3x the total weights; the question was whether the extra experts buy quality at equal decode speed.
Later reason (GGUF/seat work): Alan, 2026-10-05 09:51Z, "so we need to see what's needed for a guff run, having it with multiple seats available IF and a big IF it consolidates models it makes routing easier". And 2026-10-11 11:07Z: "I'm trying to figure out how to give kolibri1 a seat".

Open questions

The full Kolibri Q4_0 never ran the six extraction suites (enrichment, epistemics, grounding, tagger, tagger_nullable, needle) nor the -np 4 gauntlet; the Sakura pruned build did, so Sakura-vs-Kolibri comparisons only cover seven shared suites.
A low-effort arm of the 624-task MLX study (4-5 h) was never run (a lab note, "Not done").
No run at the vendor sampling settings.
The M5 mini plan (Kolibri as on-demand brain on a second host) is not on record as executed.
Whether the second Q8_0 (83 GB) is still wanted now that Sakura covers the swap seat: not on record.

The model in detail

field	value
vendor	Aleph Alpha; "Aleph Alpha, released 3 Oct 2026 (yesterday)" in the assistant's brief of 2026-10-04
upstream repo	`Aleph-Alpha/Kolibri-1` (FP8 weights on HF); revision not on record (would have been in the HF fetch of 2026-10-04)
architecture	MoE, 78B total / 3.46B active per token; 50 layers, hidden 2560; 384 routed experts per layer, 6 selected, 1 shared; 48 attention heads, 4 KV heads; "4:1 causal sliding-window GQA to full causal GQA" (40 of 50 layers keep a 513-token window)
context	262,144 native, 1,048,576 by extrapolation
behaviour	reasoning model (`reasoning_effort` low/medium/high/none, `enable_thinking`), Hermes-style tool calls, EN/DE, Apache 2.0; 20T pre-training tokens, SFT then 1,000 RL steps over 38 environments (retrieval, abstention, tool calling)
vendor sampling	temp 1.0 / top_p 0.97 / top_k 128 — not used here: the harness runs temp 0 for comparability
GGUF arch string	`kolibri1`; stock llama.cpp has no such arch, so every GGUF run needs a patch (669-line `kolibri1-llama.cpp.patch` shipped in `Hob-forge/Kolibri-1-GGUF`)
runtimes used	(1) MLX: release mlx-lm 0.32.0 on mlx 0.32.3 plus the `run.py`/`kolibri1.py` shim from `velaia/Kolibri-1-MLX-4bit`; (2) llama.cpp worktree `llama.cpp-kolibri` (two builds, see below)

Local llama.cpp builds (both patched with the Hob-forge patch): build/ = 0.5.0-dev (build 11396, commit 2e7c58c54) with the patch applied uncommitted in the worktree; build-master/ = 0.6.0-dev (build 11533, commit 769ced466), the patch rebased onto upstream 6184e92c5. build/ was later corrupted by a rebuild into the same directory and is documented as broken. The patch is exported as fleet-roster/runtimes/llama.cpp/patches/kolibri1.patch (commit ecc25d4). The other community patch (Eliasfpv28, used by the Q3_K_S and the InsidiousFiddler "XL" quants) hard-codes different router gating and is not interchangeable; only the Hob-forge GGUFs carry expert_gating_func.