Suite · chainwalk← Suites

Long horizon chaining

How deep can the model chain tool calls before it loses its place?

Cases: 2
Scored runs: 29
Models: 29
Top score: 50.0%

SpecificationMethod · grading · caveat

Method: Each step appends a call and its result to the history. By step 200 the model is reading a long transcript of its own work. The suite records how far it gets and whether it deviates.
Grading: Depth reached against depth targeted, plus a check on arithmetic carried across steps.
Caveat: Chaining and arithmetic separate cleanly. Models walk 200 steps without deviating and still get the running total wrong.
Version: chainwalk/3
Cases: 2

Recorded runs29 runs · sorted by accuracy

Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.

Top of the suite Gemma 4 12B 50.0% 95% interval 9.5–90.5

Model	Score	95% interval	Passed	Runtime · profile	Recorded
Gemma 4 12Bgemma4-12b	50.0%	9.5–90.5	Passed 1/2	llama.cppagentic-fast	2026-08-24
gemma4-12b-r1	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppagentic-fast	2026-08-24
gemma4-12b-r2	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppagentic-fast	2026-08-24
gemma4-12b-r3	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppagentic-fast	2026-08-24
k2-36b-ctx128k-q8	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-09
k2-36b-ctx32k	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-09
K2-Horizon-3.7B (IFM)k2-horizon-3.7b	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-09
k2-horizon-7b	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-09
qwen-agentworld-35b	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-07
qwen3-next-80b-ag	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-07
qwen3-next-80b-reap-ag	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppagentic-fast	2026-09-08
qwen3.6-35b-ag	50.0%	9.5–90.5overlaps the top interval	Passed 1/2	llama.cppextraction	2026-09-07
Bonsai-27B (Q1_0)bonsai-27b-1bit	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-15
bonsai-27b-mlx2-m4pro	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	mlxagentic-fast	2026-09-14
Bonsai-8B (Q1_0)bonsai-8b-1bit	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-14
BTL-4 Compact (IQ2_XXS)btl-4	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-15
BTL-4 (Q4_K_M)btl-4-q4km	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-15
flashnext-4.27bpw-m64	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-08
Gemma 4 E2Bgemma4-e2b	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e2b-r1	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e2b-r2	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e2b-r3	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e4b	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e4b-r1	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e4b-r2	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
gemma4-e4b-r3	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
Qwen3.8-9B-Distill (Q8_0)qwen3.8-9b-distill	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-14
qwen3.8-iq1s	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-08-24
Ternary-Bonsai-27B (prism-ml)ternary-bonsai-27b	0.0%	0.0–65.8overlaps the top interval	Passed 0/2	llama.cppagentic-fast	2026-09-14

Run log32 logged runs of this suite

Every logged run, newest first (32)

Finished	Model	Runtime · profile	Score	Coverage	Grader · dataset · config
2026-09-15 09:34	btl-4-q4km	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 9b7c71d675210c91
2026-09-15 08:22	btl-4	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 8dbbe23a8427e3bc
2026-09-15 00:18	bonsai-27b-1bit	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 315259486d2b1e0e
2026-09-14 23:18	bonsai-8b-1bit	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 6d222e16c742210f
2026-09-14 20:51	bonsai-27b-mlx2-m4pro	mlxagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 16fb378c7bf6e96a
2026-09-14 18:39	ternary-bonsai-27b	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 078f17fd21ba7d57
2026-09-14 15:48	qwen3.8-9b-distill	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 43d9a11df6da4927
2026-09-09 22:00	k2-36b-ctx128k-q8	not recordedextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c b3aa4c4b6efce314
2026-09-09 21:28	k2-36b-ctx32k	not recordedextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c b3aa4c4b6efce314
2026-09-09 17:16	k2-horizon-3.7b	not recordedextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 497e8e74501f9a9e
2026-09-09 16:38	k2-horizon-7b	not recordedextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c ce69ff4b8656191d
2026-09-08 19:59	flashnext-4.27bpw-m64	not recordedagentic-fast	0.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 17c29ba75eeadc76
2026-09-08 19:10	qwen3-next-80b-reap-ag	not recordedagentic-fast	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c ccae884465959abc
2026-09-07 20:19	qwen-agentworld-35b	llama.cppextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c c4305e58879e2b40
2026-09-07 20:19	qwen3-next-80b-ag	llama.cppextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 3a96fdcc21129ae7
2026-09-07 20:19	qwen3.6-35b-ag	llama.cppextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c ea9f2c30d07a149b
2026-09-07 19:24	qwen3.6-35b-ag	llama.cppextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c ea9f2c30d07a149b
2026-09-07 19:12	qwen3-next-80b-ag	llama.cppextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c 3a96fdcc21129ae7
2026-09-07 19:00	qwen-agentworld-35b	llama.cppextraction	50.0%	Coverage 2/2	g f4bfcf6f040b0978d 129a9e74408c8919c c4305e58879e2b40
2026-08-24 03:02	qwen3.8-iq1s	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c e2e5d225bbfe60d6
2026-08-24 03:00	gemma4-12b-r3	llama.cppagentic-fast	50.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 2cbb0e64811ac557
2026-08-24 02:45	gemma4-e4b-r3	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 7e97acd7556a3909
2026-08-24 02:42	gemma4-e2b-r3	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 24f4c1683cfb7a85
2026-08-24 02:38	gemma4-12b-r2	llama.cppagentic-fast	50.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c e1a3cdf8d371cae6
2026-08-24 02:23	gemma4-e4b-r2	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c ec61f0c3716fc3f8
2026-08-24 02:20	gemma4-e2b-r2	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 277579259f233168
2026-08-24 02:14	gemma4-12b-r1	llama.cppagentic-fast	50.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 97645e44ce72b737
2026-08-24 01:51	gemma4-e4b-r1	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 7ef98c4a8fe4369b
2026-08-24 01:48	gemma4-e2b-r1	llama.cppagentic-fast	0.0%	Coverage 2/2	g d31d3c4c6285617fd 129a9e74408c8919c 92c0ac5ca29b9cdc
2026-08-24 01:00	gemma4-12b	llama.cppagentic-fast	50.0%	Coverage 2/2	g 48abd22e0117fddad 129a9e74408c8919c b4834ee71de7e834
2026-08-24 00:45	gemma4-e4b	llama.cppagentic-fast	0.0%	Coverage 2/2	g 48abd22e0117fddad 129a9e74408c8919c 8e18e895c23f4485
2026-08-24 00:42	gemma4-e2b	llama.cppagentic-fast	0.0%	Coverage 2/2	g 48abd22e0117fddad 129a9e74408c8919c eef56b755300de59

Open in the run log →

Findings2 that name this suite