One function
_extracted_arxiv_ids in allenai/vla-evaluation-harness
The author described this change as “leaderboard: add newly-reviewed paper results across four benchmarks, and fix a coverage under-count (#94)”. It counts as a record because the check below fails on the code as it stood at d8e4403f8 and passes on ac8b6943f, with nothing else changed between the two runs.
Fix saved2026-07-09
Sharing licenceApache-2.0 · LICENSE
Change size+11 −6
What the code was meant to do, written into the code itself as a docstring
All arxiv IDs with an extraction record (cache preferred, packed fallback).
The change
| 1 | 1 | def _extracted_arxiv_ids() -> set[str]: | |
| 2 | - | """All arxiv IDs with an extraction record (cache preferred, packed fallback).""" | |
| 3 | - | if EXTRACTIONS_CACHE_DIR.exists(): | |
| 4 | - | return {p.stem for p in EXTRACTIONS_CACHE_DIR.glob("*.json")} | |
| 2 | + | """All arxiv IDs with an extraction record — union of the packed file and the cache dir. | |
| 3 | + | ||
| 4 | + | Either source alone under-counts: a clone that extracted a single paper has a | |
| 5 | + | one-entry cache sitting next to the full packed file. | |
| 6 | + | """ | |
| 7 | + | ids: set[str] = set() | |
| 5 | 8 | if EXTRACTIONS_PACKED_PATH.exists(): | |
| 6 | 9 | try: | |
| 7 | - | return {rec["arxiv_id"] for rec in json.loads(EXTRACTIONS_PACKED_PATH.read_text()) if rec.get("arxiv_id")} | |
| 10 | + | ids |= {rec["arxiv_id"] for rec in json.loads(EXTRACTIONS_PACKED_PATH.read_text()) if rec.get("arxiv_id")} | |
| 8 | 11 | except json.JSONDecodeError: | |
| 9 | - | return set() | |
| 10 | - | return set() | |
| 12 | + | pass | |
| 13 | + | if EXTRACTIONS_CACHE_DIR.exists(): | |
| 14 | + | ids |= {p.stem for p in EXTRACTIONS_CACHE_DIR.glob("*.json")} | |
| 15 | + | return ids |
The check that tells the two apart
fail→pass·leaderboard/tests/test_scan_extracted_ids.py::test_sparse_cache_does_not_hide_packed
Check file leaderboard/tests/test_scan_extracted_ids.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before itd8e4403f8821e08a583ac9cde99ca9c544949a90
Broken version dated2026-07-08
Moduleleaderboard.scripts.scan
Units changed_extracted_arxiv_ids
Fingerprint66834d3ac63d9748
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.