385 records from 157 projects have never been published
They were mined and verified by the same process as the 896 records that are public, on the same day, from the same crawl. The only difference is that nobody outside this machine has seen them, which is the only property that matters for measuring a model honestly.
A public evaluation set stops measuring anything once it is public
Published records end up in training data, in retrieval indexes and in the prompts people paste into chat windows. That is not misconduct by anyone in particular, it is simply what happens to text on the internet, and it means a score on a public set drifts upward over time for reasons that have nothing to do with a model getting better at fixing bugs.
The split is by project rather than by record, and permanent. Every record from a given project sits entirely on one side, so nothing in the published half can be used to infer what is in the withheld half — not the projects, not the kinds of bug, not the distribution of fixes. The file that records the assignment names every withheld project and has never left the machine that produced it.
What it is useful for
Measuring a model or an agent against records it cannot have memorised, and checking that against the number the same system scores on the public half. A large gap between the two is itself the finding, and it is not a measurement anybody can make with a public benchmark alone.
The public half stays free under CC BY 4.0 and stays published, so nothing here is taken away from anyone. What is published stays published.
Getting access
Write to andy@goldset.dev and say what you are evaluating and roughly how often. It is worth being direct about where this stands: the held-out set exists and is verified, and the terms for using it are not settled. If you tell me what you actually need, that is the thing most likely to decide them.
If you want to see the shape of the data first, every published record is browsable under records, the method is written up in the datasheet, and the results page shows four models scored against a sample of it.