Datasheet

Every figure in this document is generated from oracles.jsonl by make_datasheet.py, so it cannot drift away from the data it describes. Snapshot taken 2026-08-19: 896 records from 352 projects.

Claims that a reader cannot check against the corpus or against the public commits it cites have been left out of this document, including figures for mining throughput and cost. Those describe our pipeline rather than this dataset, and the pipeline is not published.

A held-out portion exists and is not published. Projects are assigned to the public release or to a held-out set, and a project's records are all on one side, so nothing here can be used to reconstruct what is withheld. This is stated plainly because a public evaluation set is contaminated the moment it is public, and a reader should know that a disjoint uncontaminated set exists rather than discovering it later. Everything released here stays released and is never withdrawn.

Motivation

Why was it created? Evaluating a coding agent requires knowing the right answer in a form a machine can check without a human in the loop. Goldset supplies that: real defects, the fixes their authors wrote, and the tests that tell the two apart.

Who created it? Goldset (goldset.dev). Mined from public repositories. No defect, fix or test in this corpus was authored by us.

Composition

What is an instance? A record holds the broken code, the fixed code, and a machine-runnable test that fails against the first and passes against the second.

There are 896 records drawn from 352 projects, all Python, with fixes committed between 2010-06-13 and 2026-08-17.

field	meaning
id, fingerprint	stable identity; the fingerprint deduplicates across re-runs and vendored copies
repo, repo_url, license, license_file	the source project and the licence it was read under
commit, parent, commit_url	the fix and the state it was broken in, permanently linkable
committed_at, parent_at	dates of both, for filtering against a training cutoff
subject	the author's own commit message, unedited
module, unit, units_changed	the importable path, the unit under test, and every unit the commit touched
granularity, spec_source	whether the record is one function or one file, and where its specification came from
spec	the documented behaviour of the code under test
source_before, source_after	the code before and after the fix
test_file, fail_to_pass	the test file and the exact pytest node ids that flip
validated_at, validator	audit trail

unit is empty on file-level records, where no single function was isolated.

Two granularities. 355 records isolate one changed function together with its docstring as the specification. The remaining 541 keep the whole changed file and use the commit message as the specification instead. The granularity field says which, so consumers who want only the tighter kind can filter. Defects spanning multiple source files are discovered but not extracted, so the corpus is biased toward localised, unit-testable defects.

Sampling. Not a random sample of bugs. It is the subset of real fixes that shipped a regression test in the same commit, touched a single source file, and could be reduced either to one documented unit or to one whole file whose test still flips. Use it as an evaluation set, not as an estimate of how defects are distributed in production software.

Concentration. The largest single project contributes 3% of the corpus and the ten largest contribute 20% between them, so no small group of projects dominates. An earlier version of this datasheet described the yield as following a power law, which these figures do not support. Per-project counts are published in ATTRIBUTION.md so the distribution can be checked directly.

Licence distribution.

licence	records	share
MIT	629	70.2%
Apache-2.0	117	13.1%
BSD-3-Clause	81	9.0%
BSD-2-Clause	64	7.1%
PSF-2.0	4	0.4%
ISC	1	0.1%

Confidential or offensive content? Public source code from permissively licensed projects only. No commit metadata about authors is stored: no record carries a commit author name, email or login.

Source code itself is reproduced byte for byte, and 74 records contain an email address that the project's own authors wrote into their source, usually a maintainer contact in a module header. Those addresses are already public in the repositories cited and are not extracted, indexed or collected by us, but they are present in the text, and an earlier version of this datasheet implied otherwise. If you maintain a project here and want an address removed, write to andy@goldset.dev or open an issue.

Collection

Collection is entirely by program. No language model is involved at any stage, and no human judges whether a change is a fix.

1. Read a project's commit history for changes that touched source and test files in the same commit, excluding documentation, style, refactor and revert commits.
2. Check the licence before cloning in full, so projects outside the allow-list never enter the pipeline.
3. Materialise the parent tree and the fix tree, then overlay the fix commit's test files onto the parent tree, so the same test runs against both states.
4. Run the tests the fix commit added against both trees. Keep the candidate only if they fail on the parent and pass on the fix.
5. Isolate the single changed function with its docstring where the fix allows it, and otherwise keep the whole changed file with the commit message as the specification.

Step 4 is the whole warrant. Validity is decided by execution rather than by us.

What step 4 checks, precisely. Only the tests the fix commit added are run, not the project's whole suite. A record therefore asserts that those tests fail on the parent and pass on the fix. It does not assert that the fix is globally correct, and a commit that repaired one thing while breaking another would not be caught. The benefit is that an unrelated failing test elsewhere in the file does not disqualify a genuine defect.

Independent re-verification. verify.py re-clones each project from scratch, checks out both commits, re-applies the fix commit's tests and runs them again. It reads only the published record and the project it cites, and re-derives the result from the upstream repository.

Old projects are run under an interpreter that can import them, so a record from 2016 is not judged by whether it works on a current Python. The interpreter used is reported alongside the verdict. A record that cannot be run on a given machine is reported as skipped, never as a failure.

python3 verify.py --sample 20

Run it yourself. A SKIP means the test could not be executed on your machine, most often because a project dependency is missing, and says nothing about the record. A FAIL means a published record did not hold up and is worth an issue.

Preprocessing

Doctests are removed from every spec, including the expected output beneath each example, so a model cannot read the answer out of the documentation. An earlier version stripped only the prompt lines and left the expected output stranded, which was corrected and the affected records rewritten. Source is otherwise byte-identical to upstream. Nothing is paraphrased, reformatted or model-generated.

Uses

Intended. Evaluating code understanding and defect repair in models and agents, measuring the contribution of harness and scaffolding rather than the model alone, and research on automated program repair.

Out of scope. This is an evaluation corpus. It is not a training set, and it is not a representative sample of production defects.

What passing does not mean. A model that makes the recorded test pass has made the recorded test pass. It has not been shown to have fixed the program correctly.

Contamination. Every record comes from a public repository and may already appear in a model's pre-training data. Every record carries committed_at for the fix and parent_at for the broken state, so the corpus can be cut at any training cutoff. Report the date window you evaluated on.

window	records	share
Committed in 2026	228	25%
Committed in 2025 or later	296	33%
Committed in 2024 or later	359	40%
Committed before 2024	537	60%

Because mining is continuous rather than a single snapshot, the portion of the corpus that postdates any given cutoff grows over time instead of decaying.

Distribution and licensing

Two layers, deliberately separated.

Source excerpts remain under the licence of the project they came from. Every record names its licence and the file that licence was read from, and links the exact upstream commit. Attribution for every project is in ATTRIBUTION.md, regenerated whenever the corpus changes, and the full text of each project's licence as it stood at the commit cited is included under licenses/, since MIT, BSD and Apache-2.0 condition redistribution on the notice travelling with the code rather than on a link to it. A project whose licence text could not be retrieved has its records withheld.

The added layer, meaning the validation results, the schema, the fingerprints and the selection and arrangement of records, is released under CC BY 4.0.

Only licences on an explicit allow-list are shipped. Anything ambiguous is withheld rather than guessed at.

If you maintain a project here and want it removed, write to andy@goldset.dev or open an issue, and it will be dropped, no questions asked. The address is given as well as the issue tracker so the request works even if you would rather not do it in public.

Maintenance

Mined continuously. The corpus is append-only and deduplicated by fingerprint. Records are not silently edited: a correction is a new record, except where a defect in our own extraction required rewriting a field, which is recorded below.

Errata. Defects found in the corpus itself are published rather than quietly fixed.

• Specs derived from docstrings originally retained the expected output of doctests after the prompt lines were stripped. For functions defined by their examples this disclosed part of the answer. Corrected 2026-08-18, and 84 affected records were rewritten in place.
• ATTRIBUTION.md credited only 197 of the then 386 projects for a period, because it was regenerated on the mining host rather than where the corpus was assembled. Corrected, and it is now rewritten on every change to the corpus.
• A fix that changes only an error message, while both versions raise the same exception type, cannot always be distinguished by a single test expression.
• "No known defect" means the project's own tests passed at that commit. It is not a proof of correctness.