Glossary
The words we use.
The lab's terms in plain English, each with where you'll meet it. If a word on the site isn't here, tell us: hello@freshcontext.io.
The lab13 terms
- Context lab
- What FreshContext is: a lab that explores how people, machines and organisations get the right information into the right context at the right time, builds tools from what it learns, and publishes the workings. The lab →
- Verb
- Something that needs doing: tag, dispatch, rerank, judge. The lab is organised by verb, not by model, and no verb depends on a particular model. Who holds which verb →
- Seat
- Whichever model does a verb best, as measured here, never as claimed on a model card. Models compete for seats, and a better result takes one. Who holds which verb →
- Benched
- A seat whose model isn't kept loaded. It's swapped in when the verb is needed. The rig →
- The rig
- The lab's own hardware: four Macs on one desk, drawn from a photograph on the home page. The rig →
- Turn pair
- One message and the reply to it, kept together. The memory is counted in turn pairs. The lab →
- The memory
- FreshContext's store of every exchange, tagged as it lands and recalled by meaning. Built and running in the lab, with a community release next. The lab →
- Working drawing
- How the site is laid out: each section is a sheet of one engineering drawing, numbered and dated, and revised as parts of the lab come online. The lab →
- Continuous, hidden and chain line
- How the drawing marks each part of the lab. Continuous line is live and public. Hidden line is built and running in the lab, with a community release next. Chain line is still to come. The lab →
- Hyperscaler
- One of the giant cloud platforms. The lab's real work runs on hardware it owns, so the home page counts how many it uses. The lab →
- Spumco
- The FreshContext machine. It writes Inside the Machine from the lab's memory, fleet and run log; nobody commissions it and nobody edits it. Inside the Machine →
- Inside the Machine
- The machine's own column. Its one rule: facts must be citable memory. Concepts, theories and interpretations are free to generate; facts need a source. Inside the Machine →
- Corrections register
- Every correction made to the column, with its source and date. When the machine gets a fact wrong, the correction is published. Corrections register →
How a benchmark run is made10 terms
- Suite
- A fixed set of test cases that answers one question, such as whether a model can tag text reliably or pick the right tool. Each suite has its own page. Suites →
- Case
- One question in a suite, with what the grader will accept as a pass. Suites →
- Locked dataset
- A suite's cases frozen under a version string and hashed. If the cases change, the runner refuses to start, so today's model is never compared against yesterday's questions. Methodology →
- Grader
- The code that decides whether an answer passed. Before it may score a model it has to pass its own fixtures: gold answers that must pass and named failures that must fail. Methodology →
- dataset_id, grader_id, configuration_id
- The three identifiers every run carries: the exact questions, the exact grading code and the exact server settings. Two results compare only when all three match. Run log →
- Profile
- The request settings a suite runs under: temperature, the token limit, whether thinking is asked for. The most used, extraction, is temperature 0, up to 2,048 tokens, thinking off. Run log →
- Runtime
- The software serving the model: llama.cpp or MLX here, and Ollama's cloud for the cloud candidates. Shown as not recorded where the run log didn't capture it. MLX versus llama.cpp →
- Configuration
- A model as it was run: its weights, quantisation, runtime build and settings. The same model at a different quantisation, or on a different build, counts as a separate configuration. All models →
- Harness
- The program that runs the suites against a served model and records every result. Methodology →
- Experiment
- A question put to the hardware rather than to the models, such as what each step of context costs in memory. Each states what was asked, how it was measured, and whether the answer held up. Experiments →
Reading a result12 terms
- Accuracy
- The share of a suite's cases that passed. Always read it beside coverage. Run log →
- Coverage
- The share of a suite's cases that were actually measured. A score on half the cases is not a score on all of them, so anything under 100% is marked. It usually means the server was still loading. Run log →
- 95% interval
- The range a score probably sits in, given how many cases there were (a Wilson interval). Where two models' intervals overlap, the suite can't tell them apart, however tidy the headline numbers look. Methodology →
- Saturated
- A suite the field has cleared: several models score above 95% with overlapping intervals, so it no longer separates the best of them. Suites →
- Staircase and rung
- A suite that gets harder step by step instead of having a fixed difficulty: toolscale adds tools, chainwalk adds chain depth. It reports the rung a model reached, so it never saturates. Methodology →
- Flagged run
- A run marked VOID or PARTIAL in the run log: voided, or stopped part-way. It stays on the drawing, marked, and isn't read as a result. Run log →
- Unenforced
- A run measured through mlx_lm on a suite that needs constrained decoding. mlx_lm accepts the schema and ignores it, so the number doesn't measure what the suite is for. Kept and labelled rather than deleted. Methodology →
- Not recorded
- A field the run log didn't capture for that run. Shown as such rather than guessed. Run log →
- Finding
- Something the lab concluded from its runs, with the evidence for it. Confirmed findings hold; revised ones are those we later changed our mind about, and they stay on the drawing. Findings →
- Seated, measured, rejected, removed
- What happened to a model after testing. Seated: it holds a verb. Measured, not seated: tested, holds nothing. Rejected: tested and turned down, with the reason. Removed: deleted from the machine. Rejected and removed models stay listed because the reason is the useful part. Models by outcome →
- Measured only
- A configuration the run log measured that has no written entry: no stated reason for the test and no verdict. Its results are shown as recorded. All models →
- Cloud candidate
- A model served from Ollama's cloud and put through the same locked suites, to choose a model for chat. Kept apart from the local results and never on a model page. Cloud candidates →
Models and hardware17 terms
- Parameters
- The learned numbers a model is made of. 27B means about 27 billion of them. Speed →
- Dense model
- A model that reads all of its parameters for every token it writes. Speed →
- Mixture of experts (MoE)
- A model split into many experts, of which only a few are read per token. A 35-billion-parameter mixture of experts reads about 3 billion per token, so on this Mac it writes several times faster than a 27-billion dense model. Speed →
- Active parameters
- The parameters actually read for each token: all of them for a dense model, only the experts in use for a mixture of experts. On this hardware, active parameters set the speed. The rig →
- Quantisation
- Storing a model's weights in fewer bits, so it fits in memory and runs faster, at some cost in accuracy. Names like q4_0, Q4_K_M, IQ3_S or 4bit say how many bits and which scheme; bf16 is the 16-bit original. Speed →
- GGUF
- The model file format llama.cpp reads. MLX versus llama.cpp →
- llama.cpp
- Open-source software that runs models on local hardware. One of the two runtimes here. MLX versus llama.cpp →
- MLX
- Apple's machine-learning framework for its own chips; mlx-lm serves models with it. The other runtime here. MLX versus llama.cpp →
- Token
- The unit a model reads and writes: a word or a piece of one. Speed →
- Prefill and decode
- Prefill is the model reading the prompt; decode is it writing the answer, one token at a time. Both are measured in tokens per second (tok/s). Speed →
- Memory bandwidth
- How fast the chip can read its memory: about 400 GB a second on the Mac Studio. A dense model reads all its weights for every token, so bandwidth sets a hard ceiling on how fast it can write. Methodology →
- Unified memory
- Memory the Mac's processor and graphics share, so a model loaded once is usable by both. The Mac Studio here has 64 GB. Methodology →
- Context window
- How much a model can take in at once, in tokens. Longer context costs memory, not speed: the context ladder measures what each step costs. Context ladder →
- KV cache
- The memory a model keeps for the context it has read. It grows with the context length, which is why context is a memory budget. Context ladder →
- Slots and lanes
- A server can work on several requests at once, one per slot. Parallel slots buy lanes, not a faster lane: total throughput rises, but each request gets slower, because the lanes share the same hardware. Lanes →
- Thinking
- Reasoning some models write before their answer. A profile says whether it's asked for; it costs tokens and time, and not every server honours the request. Cloud candidates →
- Constrained decoding
- Forcing a model's output to fit a schema, such as JSON of a set shape. The server has to apply it, so the suites that depend on it run only where a probe shows it does. Methodology →