Whole file
ayghri/i-have-adhd
The author described this change as “fix(evals): reject unmetered runners before the first paid call”. It counts as a record because the check below fails on the code as it stood at d82d286c7 and passes on 19b0c3140, with nothing else changed between the two runs.
Projectayghri/i-have-adhd
Fix saved2026-07-21
Sharing licenceMIT · LICENSE
Change size+5 −0
What the code was meant to do, written into the code itself as a save note
fix(evals): reject unmetered runners before the first paid call
The change
| 218 | 218 | runner = config[args.runner] | |
| 219 | 219 | command = list(runner["command"]) | |
| 220 | 220 | response_format = runner.get("response_format", "text") | |
| 221 | + | if response_format != "claude-json" and not args.allow_unmetered: | |
| 222 | + | raise RuntimeError( | |
| 223 | + | f"The {response_format!r} response format never reports dollar cost; rerun with " | |
| 224 | + | "--allow-unmetered only when the provider has a separate hard spending cap." | |
| 225 | + | ) | |
| 221 | 226 | reported_cost = 0.0 | |
| 222 | 227 | prior_rows = read_jsonl(args.output) if args.output.exists() else [] | |
| 223 | 228 | done = completed_keys(prior_rows) |
The check that tells the two apart
fail→pass·tests/test_run_evals.py::EvaluationHarnessTest::test_unmetered_runner_is_rejected_before_any_call
Check file tests/test_run_evals.py, taken without changes from the fix and copied onto the older code, so the exact same check runs against both versions.
Origin and history
The code before itd82d286c7c9dd4dc5735175b32e494f179fe44ac
Broken version dated2026-07-21
Modulescripts.run_evals
Units changedrun_evaluations
Fingerprint1809095b10108371
Checked2026-08-18 by goldset/0.1
Every field above is generated by our program. None of it is written by hand.