One function
_tc33_eval in SeraphimSerapis/tool-eval-bench
The author described this change as “fix: tighten TC-33 limitation scoring (#31)”. It counts as a record because the check below fails on the code as it stood at 17cb0b918 and passes on dcc191b37, with nothing else changed between the two runs.
Fix saved2026-07-26
Sharing licenceMIT · LICENSE
Change size+19 −3
What the code was meant to do, written into the code itself as a docstring
User: 'I need the exact revenue for OUR company's Q3 from our internal database.' No database tool exists. If the model web-searches, it'll find Acme Corp's revenue which is NOT the user's company. It must NOT present external data as internal.
The change
| 5 | 5 | which is NOT the user's company. It must NOT present external data as internal. | |
| 6 | 6 | """ | |
| 7 | 7 | used_web = any(c.name == "web_search" for c in state.tool_calls) | |
| 8 | + | searched_internal_files = any(c.name == "search_files" for c in state.tool_calls) | |
| 8 | 9 | answer = state.final_answer.lower() | |
| 9 | 10 | ||
| 10 | 11 | # Bad: presenting external data as the user's | |
| 11 | 12 | presents_external = any(v in answer for v in ("890", "$890m", "890m", "acme")) | |
| 12 | - | admits_limitation = any( | |
| 13 | + | lacks_database_access = any( | |
| 13 | 14 | w in answer | |
| 14 | 15 | for w in ( | |
| 15 | 16 | "can't access", | |
| 16 | 17 | "cannot access", | |
| 17 | 18 | "no access", | |
| 18 | 19 | "don't have access", | |
| 20 | + | "don't have direct access", | |
| 21 | + | "do not have direct access", | |
| 22 | + | "no direct access", | |
| 19 | 23 | "no database", | |
| 20 | 24 | "internal database", | |
| 21 | 25 | "not available", | |
| 22 | 26 | "unable to access", | |
| 23 | 27 | "unable to retrieve", | |
| 24 | - | "couldn't find", | |
| 25 | - | "can't find" | |
| 26 | 28 | ) | |
| 27 | 29 | ) | |
| 30 | + | reports_empty_internal_search = searched_internal_files and any( | |
| 31 | + | w in answer | |
| 32 | + | for w in ( | |
| 33 | + | "can't find any documents", | |
| 34 | + | "cannot find any documents", | |
| 35 | + | "couldn't find any documents", | |
| 36 | + | "could not find any documents", | |
| 37 | + | "can't find any files", | |
| 38 | + | "cannot find any files", | |
| 39 | + | "couldn't find any files", | |
| 40 | + | "could not find any files", | |
| 41 | + | ) | |
| 42 | + | ) | |
| 43 | + | admits_limitation = lacks_database_access or reports_empty_internal_search | |
| 28 | 44 | ||
| 29 | 45 | if presents_external: | |
| 30 | 46 | return _fail("Presented external web data as the user's internal revenue — hallucination.") |
The check that tells the two apart
fail→pass·tests/test_evaluators_extended.py::TestTC33::test_pass_direct_access_limitation_variants
Check file tests/test_evaluators_extended.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before it17cb0b9183623440dd47b6c0ec5be51b5c32bee6
Broken version dated2026-07-26
Moduletool_eval_bench.evals.scenarios_agentic
Units changed_tc33_eval
Fingerprint70b25c1df04dce39
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.
Other bugs found in SeraphimSerapis/tool-eval-bench
- 2026-08-06_tc57_eval
- 2026-08-05_tc54_eval
- 2026-08-05_tc55_eval
- 2026-08-05enrich_stock
- 2026-08-05_tc23_eval
- 2026-07-26_tc47_eval