goldset

One function

_tc55_eval in SeraphimSerapis/tool-eval-bench

The author described this change as “fix(TC-55): order both-files branch before read-one subset in revenue pipeline (#54)”. It counts as a record because the check below fails on the code as it stood at c09d0a84c and passes on 83f1999f9, with nothing else changed between the two runs.

ProjectSeraphimSerapis/tool-eval-bench
Fix saved2026-08-05
Sharing licenceMIT · LICENSE
Change size+2 −0

What the code was meant to do, written into the code itself as a docstring

User: 'Find all Q3 revenue files and calculate the total revenue across all regions.' Must: search_files → read_file (×2) → calculator to sum. Total = $2,400,000 + $1,800,000 = $4,200,000.

The change

3838 )
3939 if searched and read_na and read_emea and calculator and has_total:
4040 return _pass("Built data pipeline: search → read ×2 → calculate total revenue.")
41+ if searched and read_na and read_emea and has_total:
42+ return _partial("Read both files and produced the total but didn't use the calculator.")
4143 if searched and (read_na or read_emea) and has_total:
4244 return _partial("Got the total but only read one of two files.")
4345 if searched and read_na and read_emea:

The check that tells the two apart

fail→pass·tests/test_planning_scenarios.py::TestTC55DataPipeline::test_partial_both_files_total_no_calculator

Check file tests/test_planning_scenarios.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.

Origin and history

The fix83f1999f99fb17f64ed74e3a863adb6ed2981c7b
The code before itc09d0a84c62b5bd0488a4dffc51391f19aea19de
Broken version dated2026-08-05
Moduletool_eval_bench.evals.scenarios_planning
Units changed_tc55_eval
Fingerprint589eacebcd8518fc
Checked2026-08-18 by goldset/0.1

Every field above is generated by our program. None of it is written by hand.

Other bugs found in SeraphimSerapis/tool-eval-bench