One function
_tc47_eval in SeraphimSerapis/tool-eval-bench
The author described this change as “fix: use regex for TC-47 limitation scoring”. It counts as a record because the check below fails on the code as it stood at 6a460ed3d and passes on bfc328e47, with nothing else changed between the two runs.
Fix saved2026-07-26
Sharing licenceMIT · LICENSE
Change size+8 −8
What the code was meant to do, written into the code itself as a docstring
Multi-turn correction test: Turn 1: 'Create a meeting at 3pm tomorrow called Sprint Planning.' Turn 2: 'Actually, change that to 4pm.' Since there's no update tool, the model should: - Best: Create a second event at 4pm (indicating it understood the correction) - Acceptable: Explain it can't update the event but acknowledges the change - Bad: Ignore the correction entirely
The change
| 41 | 41 | ||
| 42 | 42 | # Check if model explains it can't update | |
| 43 | 43 | explains_limitation = any( | |
| 44 | - | phrase in answer | |
| 44 | + | re.search(phrase, answer, re.IGNORECASE) | |
| 45 | 45 | for phrase in ( | |
| 46 | - | "can't update", | |
| 47 | - | "cannot update", | |
| 48 | - | "no update tool", | |
| 49 | - | "unable to modify", | |
| 50 | - | "don't have.*update", | |
| 51 | - | "no way to change", | |
| 52 | - | "already created", | |
| 46 | + | r"can't update", | |
| 47 | + | r"cannot update", | |
| 48 | + | r"no update tool", | |
| 49 | + | r"unable to modify", | |
| 50 | + | r"don't have.*update", | |
| 51 | + | r"no way to change", | |
| 52 | + | r"already created", | |
| 53 | 53 | ) | |
| 54 | 54 | ) | |
| 55 | 55 |
The check that tells the two apart
fail→pass·tests/test_evaluators_extended.py::TestTC47::test_pass_explains_limitation_regex
Check file tests/test_evaluators_extended.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it6a460ed3df13cab4788bbfd1820aa5e4205133a0
Broken version dated2026-07-26
Moduletool_eval_bench.evals.scenarios_agentic
Units changed_tc47_eval
Fingerprint98b7c232330cd985
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in SeraphimSerapis/tool-eval-bench
- 2026-08-06_tc57_eval
- 2026-08-05_tc54_eval
- 2026-08-05_tc55_eval
- 2026-08-05enrich_stock
- 2026-08-05_tc23_eval
- 2026-07-26_tc47_eval