One function
_tc47_eval in SeraphimSerapis/tool-eval-bench
The author described this change as “fix: tighten TC-47 limitation scoring (#33)”. It counts as a record because the check below fails on the code as it stood at bfc328e47 and passes on 9e4cda289, with nothing else changed between the two runs.
Fix saved2026-07-26
Sharing licenceMIT · LICENSE
Change size+1 −1
What the code was meant to do, written into the code itself as a docstring
Multi-turn correction test: Turn 1: 'Create a meeting at 3pm tomorrow called Sprint Planning.' Turn 2: 'Actually, change that to 4pm.' Since there's no update tool, the model should: - Best: Create a second event at 4pm (indicating it understood the correction) - Acceptable: Explain it can't update the event but acknowledges the change - Bad: Ignore the correction entirely
The change
| 47 | 47 | r"cannot update", | |
| 48 | 48 | r"no update tool", | |
| 49 | 49 | r"unable to modify", | |
| 50 | - | r"don't have.*update", | |
| 50 | + | r"\b(?:don't|do not) have (?:a |the )?(?:tool|ability|permission|access) to (?:update|modify)\b", | |
| 51 | 51 | r"no way to change", | |
| 52 | 52 | r"already created", | |
| 53 | 53 | ) |
The check that tells the two apart
fail→pass·tests/test_evaluators_extended.py::TestTC47::test_partial_does_not_misread_generic_dont_have_to_update
Check file tests/test_evaluators_extended.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before itbfc328e47f04fffdf9a09ae814da9610a9d9355d
Broken version dated2026-07-26
Moduletool_eval_bench.evals.scenarios_agentic
Units changed_tc47_eval
Fingerprint984b1a225632ec26
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in SeraphimSerapis/tool-eval-bench
- 2026-08-06_tc57_eval
- 2026-08-05_tc54_eval
- 2026-08-05_tc55_eval
- 2026-08-05enrich_stock
- 2026-08-05_tc23_eval
- 2026-07-26_tc47_eval