Four models, 75 records each, scored by running the tests
Every model was given the same records, the specification and the broken code, and asked to return the fix. It never saw the test or the author’s answer. Its output was written back into the project and the recorded tests were run against it.
Results
| Model | Solved | Scored | pass@1 | 95% confidence |
|---|---|---|---|---|
| google/gemini-3.7-flash | 28 | 71 | 39.4% | 28.9% to 51.1% |
| deepseek/deepseek-v4-flash | 17 | 70 | 24.3% | 15.8% to 35.5% |
| openai/gpt-oss-120b | 17 | 70 | 24.3% | 15.8% to 35.5% |
| qwen/qwen3-coder-30b-a3b-instruct | 7 | 73 | 9.6% | 4.7% to 18.5% |
Records a machine could not run are reported separately and left out of the score. Nothing is scored unless the author’s own fix passes on the same machine first, so an environment failure is never counted against a model.
Of the 66 records every model attempted, 38 were solved by none
That is 58% of the sample that no model here could fix. Only 4 were solved by all four, so almost nothing in this set is a free point, and 9 were solved by exactly one model. Those last ones are where the models genuinely differ.
The spread between best and worst is 29.8%. A set where every model scores the same measures nothing, so this is the property that decides whether the corpus is worth running at all.
What this does not establish
- −The two middle models are separated by less than their confidence intervals. Treat them as tied.
- −These are four fast, inexpensive models. Nothing here measures a frontier model.
- −75 records is a small sample. Every figure carries an interval for that reason.
- −One model was re-run after empty replies from the provider were retried. The other three returned none, so the retry cannot have changed their scores.
Run it yourself with goldset eval --model <model>, or read the datasheet for how a record is made.
284 scored evaluations across 4 models.