goldset

Verified bug-fix records for evaluating coding agents

Every record is a real bug in public software, the fix its author wrote, and the test that fails before the fix and passes after it. A record is kept only once both runs have been observed, so what is published is a reproduction rather than a claim.

896
records
352
projects
$0.0048
per record
100%
permissive licences

One record

A generator in more-itertools let a stopping signal from the caller’s own function escape into it and end the loop early.

more-itertools/more-itertoolsiterate2023-04-19MIT
99 while True:
1010 yield start
11- start = func(start)
11+ try:
12+ start = func(start)
13+ except StopIteration:
14+ break
fails on 7c1b6482a, passes on cd0a3a87d·open

Method

  1. 1Find commits that changed source and tests together, in permissively licensed projects only.
  2. 2Run the author’s test against the code as it stood before the fix. It must fail.
  3. 3Run the same test against the fix. It must pass.
  4. 4Keep the changed function with its docstring, or the whole file when it does not isolate cleanly.

About one candidate in ten survives. No language model is used at any stage.

Compared with related datasets

DatasetRecordsProjectsBugs from
SWE-bench2,29012GitHub issues
SWE-bench-Live1,31993GitHub issues, 2024 onward
SWE-smith50,137128generated
Goldset896352maintainer fix commits

Figures as each project publishes them, read from the linked sources on 2026-08-18. Of our 353 projects, 294 have under 1,000 stars or under 200 forks.

What this does not show

  • −That these bugs are harder for models than generated ones. We tested that and it did not replicate.
  • −That a passing test means the program is correct. It means the recorded test passes.
  • −Anything about languages other than Python.

A further 385 records are held back and not published

A public evaluation set is contaminated the moment it is public. Everything above can be downloaded, and in time it will be read by the models it exists to measure. So projects are split in two, and a project’s records sit entirely on one side or the other. Nothing in the public release can be used to work out what is in the held-out one.

SplitRecordsProjectsStatus
Public897352published under CC BY 4.0, permanently
Held out385157not published

Mining runs continuously, so both sides grow, and the share of the corpus that postdates any given training cutoff grows with them. That is the part a fixed benchmark cannot do. Everyone else’s uncontaminated slice shrinks as models get newer, and this one refills.

If you evaluate models and want a set yours has not seen, or the same pipeline run against other languages or your own repositories, write to andy@goldset.dev. What is published here stays published and free either way.

Get the data

git clone https://github.com/andysalvo/goldset
python3 verify_corpus.py --sample 20

One JSON object per line. 355 records isolate a single function, 541 keep the whole file. Each carries the date its fix was committed, so anything inside a model’s training window can be filtered out. A Hugging Face release is being prepared.

@misc{goldset2026,
  title  = {Goldset: verified bug-fix records for evaluating coding agents},
  author = {Salvo, Andrew},
  year   = {2026},
  url    = {https://goldset.dev}
}