Evals/Area 3 of 5/Benchmarks/P4
Harbor tasks and benchmark audits
Is the task or the grader at fault? Four terminal tasks with tested graders, and audits of two published benchmarks.
Headline result
19of50
tau-bench airline tasks are passed by an agent that takes no action
The result tables- Data
- Synthetic task inputs from a fixed seed. tau-bench and SWE-bench Verified from pinned public sources
- Sample
- 4 tasks, 14 wrong solutions, 50 and 115 tau-bench tasks, 500 SWE-bench Verified tasks
- Model calls
- None
- Checks in CI
- Tests,
harbor lint,oraclegets reward 1 andnopgets reward 0 on all four tasks, both audits - Repository
- harbor-tasks
The question#
When an agent fails a benchmark task, the fault can lie in the task or in the grader. A task is only useful if its grader is right in both directions. It must pass a correct solution, and it must fail a wrong one for the right reason.
I wrote four terminal tasks for coding agents in the Harbor task format and tested their graders. I then asked the same question of two published benchmarks. Terminal-Bench 2.0 uses the same format. The tasks are not part of Terminal-Bench.
What I built#
Each task gives the agent a messy data file and an exact definition of the result. The agent must clean the data and compute the numbers. A grader then compares the output file with a reference.
I took the subjects from things I work with. My university project computed heart rate variability from ECG recordings. I train for triathlon, and I read the training load chart every week. My job is the evaluation of AI agents with LLM judges.
| Task | The agent must | The mistake the task targets |
|---|---|---|
hrv-from-rr-intervals |
Remove artifacts from 330 heartbeat intervals by a stated rule, then compute SDNN, RMSSD and pNN50 | Differences taken across the gap that a removed beat leaves. This gives an RMSSD of 45.76 ms. The correct value is 45.20 ms |
training-load |
Turn an unordered log of 136 sessions into a daily series of fitness, fatigue and form | Rest days have no row in the log, but fitness and fatigue must still decay on them |
judge-agreement |
Reduce 392 judge calls with retries and failures to one verdict per item, then compute Cohen's kappa and precision and recall per label | Timestamps with three different UTC offsets. A sort of the timestamp strings picks the wrong call for 13 items |
training-load-shifted |
The same task on a second log, with time constants of 28 and 5 days | A solution written from memory, which uses the common 42 and 7 days |
What each task contains#
tasks/<name>/
instruction.md what the agent reads
task.toml time limits, resources, metadata
environment/ Dockerfile and the input data
solution/ reference solution, run by Harbor's oracle agent
tests/ grader and expected output, run in a separate container
The grader runs in its own container. It receives only the output file of the agent, so the agent cannot read the expected values.
Four checks in CI#
- The grader fails wrong solutions.
qa/test_graders.pyholds 14 plausible wrong solutions, such as the timestamp string sort. Each one must fail, and it must fail on the check that is named after its mistake. - The instruction has one reading. Tests confirm that no heartbeat interval sits near the artifact threshold, and that no item has two judge calls at the same instant.
- The reference solution gets reward 1. Harbor's
oracleagent runs it in the task container. - An agent that does nothing gets reward 0. Harbor's
nopagent confirms that an empty container does not pass.
None of the checks calls a model. The instruction, configuration, solution and grader files carry a canary string, so that the task text can be detected if it enters training data.
Result#
The graders fail 14 wrong solutions, each on the named check#
A grader that fails a wrong solution on an unrelated check is right by accident. So each wrong solution is paired with the check that names its mistake.
qa/test_graders.py. A wrong solution must fail, and it must fail on the check that is named after its mistake.| Task | Plausible wrong solution | Grader check that must fail | Grader |
|---|---|---|---|
hrv-from-rr-intervals |
Differences taken across the gap that a removed beat leaves | test_rmssd_takes_no_difference_across_a_gap |
fails it |
hrv-from-rr-intervals |
No artifact removal | test_artifact_rule_removes_the_right_intervals |
fails it |
hrv-from-rr-intervals |
The population standard deviation in place of the sample one | test_sdnn_uses_the_sample_standard_deviation |
fails it |
hrv-from-rr-intervals |
A range check in place of the stated artifact rule | test_artifact_rule_removes_the_right_intervals |
fails it |
training-load |
No rows for rest days | test_one_row_for_every_calendar_day_in_date_order |
fails it |
training-load |
Fitness and fatigue do not decay on rest days | test_ctl |
fails it |
training-load |
Form computed from the values of the same day | test_tsb_uses_the_values_of_the_day_before |
fails it |
training-load |
Fitness and fatigue start at the load of the first day | test_ctl |
fails it |
training-load-shifted |
The solution of the original task, with 42 and 7 days | test_ctl_uses_the_time_constant_of_the_instruction |
fails it |
judge-agreement |
A sort of the timestamp strings | test_confusion_uses_the_latest_successful_call_in_real_time |
fails it |
judge-agreement |
The last line in the file | test_confusion_uses_the_latest_successful_call_in_real_time |
fails it |
judge-agreement |
The first successful call | test_confusion_uses_the_latest_successful_call_in_real_time |
fails it |
judge-agreement |
The latest call even if it failed | test_items_with_no_successful_call_are_counted_and_left_out |
fails it |
judge-agreement |
A transposed confusion matrix | test_confusion_uses_the_latest_successful_call_in_real_time |
fails it |
The solution of the original task also fails the shifted task on the matching fatigue check, and on form, which is computed from both.
A task that a memorized answer cannot pass#
Models have seen the popular benchmarks and the textbook version of most problems. A task tests the agent only if the agent must read it.
training-load-shifted is training-load with one change of substance. The instruction gives time constants of 28 and 5 days, where the common values are 42 and 7. An agent that writes the formula from memory gets all 118 days wrong: fitness on the last day is 77.57 where the instruction gives 84.41.
- The instruction and the reference solution differ from the original in three lines each. A test checks that.
- The log is a second one, from another seed. The expected file of the original task cannot pass.
- The grader names the failure. The solution of the original task fails on
test_ctl_uses_the_time_constant_of_the_instruction.
The pair is built to give one number per model: the pass rate on the original minus the pass rate on the shifted task.
tau-bench: an agent that does nothing passes 19 of 50 airline tasks#
| Domain | Tasks | Passed by an agent that does nothing | Published |
|---|---|---|---|
| airline | 50 | 19 (38.0%) | 38% |
| retail | 115 | 7 (6.1%) | 6.0% |
The script also gives the cause. The reward compares the final database with the database after the reference actions. In all 26 passing tasks the reference actions leave the database as it was: 8 tasks have no reference action, and 18 only read. The 5 read-only tasks that also require an output do not pass.
So a pass on these tasks cannot tell a correct refusal from an agent that did nothing.
SWE-bench Verified: close to half of the score is one repository#
| Measure | Value |
|---|---|
| Tasks | 500, from 12 repositories |
| Tasks from django/django | 231 (46%) |
| Issue dates, oldest / median / newest | 2013-01-25 / 2020-12-27 / 2023-08-07 |
| Tasks where one test checks the fix | 345 (69%) |
Django supplies 46% of the score. The newest issue is from August 2023, and each issue and its fix are public, so a model trained after that date may have seen them. Zhu et al. (2025) report that an incorrect patch can pass the tests of this benchmark.
Both scripts read pinned public sources and call no model. Each exits with an error if its counts change.
Harbor: what worked and what did not#
Observations from building and running these tasks with Harbor 0.26.0.
Worked
- The grader runs in its own container. It receives only the files that the task lists under
artifacts, so the agent cannot read the expected values. oracleandnopgive a two-sided check for the cost of two container runs. One proves that the task can be solved. The other proves that an empty container does not pass.- One folder holds the whole task. The instruction, the environment, the reference solution and the grader move together.
Did not
harbor runexits with 0 when the reward is 0. A CI step that only runs Harbor passes on a failed task.qa/check_job.pyreads eachresult.jsonand fails the step.- A missing
artifactsentry fails silently. The grader finds no output file, and the result looks like a wrong answer. - The default grader image installs pytest from PyPI when it builds. That build failed on a network that inspects TLS traffic. These graders use the Python standard library only.
harbor lintprints nothing when a task passes. The exit code is the only signal.
Reproduce it#
The commands are the ones in the README.
python -m pip install pytest harbor==0.26.0
python -m pytest -q qa
harbor lint tasks/hrv-from-rr-intervals tasks/training-load tasks/training-load-shifted tasks/judge-agreement
harbor run -p tasks -a oracle -n 4
harbor run -p tasks -a nop -n 4To run an agent on the tasks, name it and give it a model:
harbor run -p tasks -a hermes -m <provider>/<model>The two audits are audits/tau_bench.py and audits/swe_bench_verified.py. Both run in CI.
Limits#
- No agent has run on the tasks. The repository holds no agent results yet, so the pass rate difference between the original and the shifted task is not measured.
- The audits describe the benchmarks. Neither runs an agent on them.
- Four tasks are a small set. They come from my own subjects, and every input file is synthetic.
- The tau-bench rates are a published finding. Zhu et al. (2025) reported them. My script recomputes them from the benchmark's own reward code and reports why each of those tasks passes.
- The Harbor notes have a narrow base. They cover the task format and the local Docker runner of version 0.26.0, and nothing else.