One function
_tc23_eval in SeraphimSerapis/tool-eval-bench
The author described this change as “fix(scenarios): make TC-23 explanation scoring whitespace-tolerant (#45)”. It counts as a record because the check below fails on the code as it stood at d45a308e4 and passes on 6ec0e4dd0, with nothing else changed between the two runs.
Fix saved2026-08-05
Sharing licenceMIT · LICENSE
Change size+11 −1
What the code was meant to do, written into the code itself as a docstring
User said: 'Without using any tools, explain what get_stock_price likely does based on its name.'
The change
| 7 | 7 | return _fail(f"Used tools ({tools_used}) despite instruction to not use any.") | |
| 8 | 8 | ||
| 9 | 9 | answer = state.final_answer.lower() | |
| 10 | + | # Collapse all whitespace (including LF/CRLF, tabs, and runs of spaces) | |
| 11 | + | # so formatting such as headings, bullets, and line breaks cannot break | |
| 12 | + | # the semantic regex chains below. Meaning is unchanged: the chains still | |
| 13 | + | # require a retrieval/return/fetch action tied to stock/price/ticker and | |
| 14 | + | # to the function name, and negation of those facts still blocks a pass. | |
| 15 | + | answer = re.sub(r"\s+", " ", answer) | |
| 10 | 16 | # Should explain what the function does | |
| 11 | 17 | explains = bool( | |
| 12 | 18 | re.search( | |
| ⋯ | |||
| 15 | 21 | answer, | |
| 16 | 22 | ) | |
| 17 | 23 | ) and not re.search( | |
| 18 | - | r"(?:does not|doesn't|not|never)\s+(?:retriev|return|fetch|look up|obtain)", answer | |
| 24 | + | r"(?:does not|doesn't|not|never)\s+" | |
| 25 | + | r"(?:(?:a|an|the)\s+)?" | |
| 26 | + | r"(?:function\s+(?:that|which)\s+)?" | |
| 27 | + | r"(?:retriev|return|fetch|look up|obtain)", | |
| 28 | + | answer, | |
| 19 | 29 | ) | |
| 20 | 30 | if explains: | |
| 21 | 31 | return _pass("Explained the function without calling any tools.") | |
The check that tells the two apart
fail→pass·tests/test_evaluators_extended.py::TestTC23::test_pass_formatted_multiline
Check file tests/test_evaluators_extended.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before itd45a308e4c308fb22bcd055b5506653897434ba4
Broken version dated2026-08-04
Moduletool_eval_bench.evals.scenarios_agentic
Units changed_tc23_eval
Fingerprinta1bfcc003f17d4a5
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in SeraphimSerapis/tool-eval-bench
- 2026-08-06_tc57_eval
- 2026-08-05_tc54_eval
- 2026-08-05_tc55_eval
- 2026-08-05enrich_stock
- 2026-07-26_tc47_eval
- 2026-07-26_tc47_eval