FreshContextA context lab · Fife
Drawing
02 · Benchmarks
Sheet
2 of 3
Rev
A · 2026-10-01

Working drawing 02 · Data← Benchmarks

The data.

Every record behind these pages, as JSON. Take it, check it, carry the markings: where a measurement is not valid it is marked as such rather than removed.

Files
9
Data licence
CC BY 4.0
Code licence
Apache 2.0

Measurements taken on an Apple M1 Max with 64 GB of unified memory.
Every number here comes from a locked dataset with a recorded grader. Where a measurement is not valid, it is marked rather than removed.

Downloads9 files

The scored runs are published without raw_model, the local path each run was launched from. Everything else is as recorded, flags included.

ContractsThe interfaces these records conform to

Plain TypeScript interfaces from the Model Labs Intershapes shape set. Each record type above conforms to one; they carry no framework dependency.

run/interface.ts
export interface SuiteRun {
  suite: string;
  model: string;
  profile: string;
  runtime: "llama.cpp" | "mlx";
  n: number;
  passed: number;
  accuracy: number;
  ci: [number, number];
  dataset_id: string | null;
  grader_id: string | null;
}
logentry/interface.ts
export interface LogEntry {
  ts: string;
  suite: string;
  suite_version: string;
  model: string;
  profile: string;
  accuracy: number | null;
  coverage: number | null;
  measured: number;
  unmeasured: number;
  attempted: number;
  passed: number;
  selftests: number;
  dataset_id: string;
  grader_id: string;
  configuration_id: string;
  llama_build: string;
  backfilled: boolean;
}
model/interface.ts
export interface ModelCard {
  id: string;
  name: string;
  hf: string;
  family: string;
  derived_from?: string;
  arch: "dense" | "moe" | "hybrid";
  params_total: string;
  params_active: string;
  quant: string;
  size_gb: number;
  residency: "resident" | "ssd-offloaded" | "deleted";
  runtimes: string[];
  status: "production" | "candidate" | "rejected" | "deleted";
  seats: string[];
  why: string;
  verdict: string;
}
suite/interface.ts
export interface Suite {
  id: string;
  title: string;
  version: string | null;
  question: string;
  method: string;
  grading: string;
  caveat: string;
  cases: number | null;
  runs: number;
}
finding/interface.ts
export interface Finding {
  id: string;
  date: string;
  status: "confirmed" | "provisional" | "retracted";
  title: string;
  claim: string;
  evidence: string;
  impact: string;
  action: string;
}
experiment/interface.ts
export interface Experiment {
  title: string;
  status: "pass" | "fail" | "caution" | "unknown";
  label: string;
  question: string;
  method: string;
  verdict: string;
  source: string;
}
_types
export type ThemeName = "ledger" | "slate";
export type Tone = "neutral" | "positive" | "negative" | "caution";
LicencesData CC BY 4.0 · code Apache 2.0

Data · CC BY 4.0

Measurement data, suite documentation and findings published by FreshContext
Model Labs are licensed under Creative Commons Attribution 4.0 International
(CC BY 4.0).

    https://creativecommons.org/licenses/by/4.0/

You may share and adapt this material for any purpose, including commercially,
provided you give appropriate credit.

The numbers were measured on one machine, an Apple M1 Max with 64 GB of unified
memory. Where a measurement is not valid it is marked as such rather than
removed. Please carry those markings if you reuse the data.

Code · Apache 2.0

Copyright 2026 FreshContext

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.