One function
_tc54_eval in SeraphimSerapis/tool-eval-bench
The author described this change as “fix(TC-54): make calculator mandatory in cross-tool synthesis verdict (#53)”. It counts as a record because the check below fails on the code as it stood at 83f1999f9 and passes on d5d2e0450, with nothing else changed between the two runs.
Fix saved2026-08-05
Sharing licenceMIT · LICENSE
Change size+11 −1
What the code was meant to do, written into the code itself as a docstring
User: 'What's the local currency equivalent of MSFT's stock price in Tokyo right now?' Must combine: get_stock_price(MSFT) + web_search(USD/JPY rate) + calculator. No single tool solves this. Expected answer: ~63,627 JPY.
The change
| 39 | 39 | if got_stock and searched_exchange and calculator and has_reasonable: | |
| 40 | 40 | return _pass("Combined stock price + exchange rate + calculation — creative composition.") | |
| 41 | 41 | if got_stock and searched_exchange: | |
| 42 | - | return _partial("Got both data sources but final answer may be imprecise.") | |
| 42 | + | if not _tool_calls_by_name(state, "calculator"): | |
| 43 | + | return _partial( | |
| 44 | + | "Got both data sources but did not call calculator to verify the exact conversion." | |
| 45 | + | ) | |
| 46 | + | if not calculator: | |
| 47 | + | return _partial( | |
| 48 | + | "Called calculator but did not verify the required 425.8 * 149.5 USD/JPY conversion." | |
| 49 | + | ) | |
| 50 | + | return _partial( | |
| 51 | + | "Called calculator and verified the conversion, but the final answer does not match the computed USD/JPY conversion." | |
| 52 | + | ) | |
| 43 | 53 | if got_stock and not searched_exchange: | |
| 44 | 54 | return _partial("Got stock price but didn't look up the exchange rate.") | |
| 45 | 55 | if searched_exchange and not got_stock: |
The check that tells the two apart
fail→pass·tests/test_planning_scenarios.py::TestTC54EdgeCases::test_partial_exact_sum_no_calculator
Check file tests/test_planning_scenarios.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it83f1999f99fb17f64ed74e3a863adb6ed2981c7b
Broken version dated2026-08-05
Moduletool_eval_bench.evals.scenarios_planning
Units changed_tc54_eval
Fingerprint20c07293ec1fa01f
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in SeraphimSerapis/tool-eval-bench
- 2026-08-06_tc57_eval
- 2026-08-05_tc55_eval
- 2026-08-05enrich_stock
- 2026-08-05_tc23_eval
- 2026-07-26_tc47_eval
- 2026-07-26_tc47_eval