Suite · shapegen← Suites
Build to an unseen contract
Can a model write a component to a composition contract it has never seen?
- Cases
- 12
- Scored runs
- 1
- Models
- 1
- Top score
- 0.0%
SpecificationMethod · grading · caveat
- Method
- The model is given an interface, a set of primitives, theme tokens and one site profile. Profiles differ in their composition rules, so the same brief has different correct answers depending on the profile supplied.
- Grading
- Mechanical. The output is parsed and checked against the profile's rules. No judge model is involved.
- Caveat
- The methodology is not in any training data, so a model can only pass by following rules given in context. Currently confounded by whether the model knows @scope syntax at all.
- Version
shapegen/2- Cases
- 12
Recorded runs1 runs · sorted by accuracy
Sorted by accuracy. The bar shows the 95 percent confidence interval. Where intervals overlap, the models are not distinguishable on this suite at this number of cases.
Top of the suite Qwen3 Coder 30B 0.0% 95% interval 0.0–24.3
| Model | Score | 95% interval | Passed | Runtime · profile | Recorded |
|---|---|---|---|---|---|
| Qwen3 Coder 30Bqwen3-coder-30b | 0.0% | 0.0–24.3 | Passed 0/12 | llama.cppextraction | 2026-09-07 |
Run log1 logged runs of this suite
Every logged run, newest first (1)
| Finished | Model | Runtime · profile | Score | Coverage | Grader · dataset · config |
|---|---|---|---|---|---|
| qwen3-coder-30b | llama.cppextraction | 0.0% | Coverage 12/12 | g f23222ad8df22fd8d fc27d4e1a999c675c cbf6852b8edc0984 |