Sara Ghorayeb

Evals/Area 3 of 5/Benchmarks/P4

Harbor tasks and benchmark audits

Is the task or the grader at fault? Four terminal tasks with tested graders, and audits of two published benchmarks.

Headline result

19of50

tau-bench airline tasks are passed by an agent that takes no action

The result tables
Data
Synthetic task inputs from a fixed seed. tau-bench and SWE-bench Verified from pinned public sources
Sample
4 tasks, 14 wrong solutions, 50 and 115 tau-bench tasks, 500 SWE-bench Verified tasks
Model calls
None
Checks in CI
Tests, harbor lint, oracle gets reward 1 and nop gets reward 0 on all four tasks, both audits
Repository
harbor-tasks

The question#

When an agent fails a benchmark task, the fault can lie in the task or in the grader. A task is only useful if its grader is right in both directions. It must pass a correct solution, and it must fail a wrong one for the right reason.

I wrote four terminal tasks for coding agents in the Harbor task format and tested their graders. I then asked the same question of two published benchmarks. Terminal-Bench 2.0 uses the same format. The tasks are not part of Terminal-Bench.

What I built#

Each task gives the agent a messy data file and an exact definition of the result. The agent must clean the data and compute the numbers. A grader then compares the output file with a reference.

I took the subjects from things I work with. My university project computed heart rate variability from ECG recordings. I train for triathlon, and I read the training load chart every week. My job is the evaluation of AI agents with LLM judges.

The four tasksEvery input file is synthetic and written from a fixed seed. No file holds real health data or real training data.
Task The agent must The mistake the task targets
hrv-from-rr-intervals Remove artifacts from 330 heartbeat intervals by a stated rule, then compute SDNN, RMSSD and pNN50 Differences taken across the gap that a removed beat leaves. This gives an RMSSD of 45.76 ms. The correct value is 45.20 ms
training-load Turn an unordered log of 136 sessions into a daily series of fitness, fatigue and form Rest days have no row in the log, but fitness and fatigue must still decay on them
judge-agreement Reduce 392 judge calls with retries and failures to one verdict per item, then compute Cohen's kappa and precision and recall per label Timestamps with three different UTC offsets. A sort of the timestamp strings picks the wrong call for 13 items
training-load-shifted The same task on a second log, with time constants of 28 and 5 days A solution written from memory, which uses the common 42 and 7 days

What each task contains#

tasks/<name>/
  instruction.md      what the agent reads
  task.toml           time limits, resources, metadata
  environment/        Dockerfile and the input data
  solution/           reference solution, run by Harbor's oracle agent
  tests/              grader and expected output, run in a separate container

The grader runs in its own container. It receives only the output file of the agent, so the agent cannot read the expected values.

Four checks in CI#

  1. The grader fails wrong solutions. qa/test_graders.py holds 14 plausible wrong solutions, such as the timestamp string sort. Each one must fail, and it must fail on the check that is named after its mistake.
  2. The instruction has one reading. Tests confirm that no heartbeat interval sits near the artifact threshold, and that no item has two judge calls at the same instant.
  3. The reference solution gets reward 1. Harbor's oracle agent runs it in the task container.
  4. An agent that does nothing gets reward 0. Harbor's nop agent confirms that an empty container does not pass.

None of the checks calls a model. The instruction, configuration, solution and grader files carry a canary string, so that the task text can be detected if it enters training data.

Result#

The graders fail 14 wrong solutions, each on the named check#

A grader that fails a wrong solution on an unrelated check is right by accident. So each wrong solution is paired with the check that names its mistake.

The 14 wrong solutions and the check that each must failFrom qa/test_graders.py. A wrong solution must fail, and it must fail on the check that is named after its mistake.
Task Plausible wrong solution Grader check that must fail Grader
hrv-from-rr-intervals Differences taken across the gap that a removed beat leaves test_rmssd_takes_no_difference_across_a_gap fails it
hrv-from-rr-intervals No artifact removal test_artifact_rule_removes_the_right_intervals fails it
hrv-from-rr-intervals The population standard deviation in place of the sample one test_sdnn_uses_the_sample_standard_deviation fails it
hrv-from-rr-intervals A range check in place of the stated artifact rule test_artifact_rule_removes_the_right_intervals fails it
training-load No rows for rest days test_one_row_for_every_calendar_day_in_date_order fails it
training-load Fitness and fatigue do not decay on rest days test_ctl fails it
training-load Form computed from the values of the same day test_tsb_uses_the_values_of_the_day_before fails it
training-load Fitness and fatigue start at the load of the first day test_ctl fails it
training-load-shifted The solution of the original task, with 42 and 7 days test_ctl_uses_the_time_constant_of_the_instruction fails it
judge-agreement A sort of the timestamp strings test_confusion_uses_the_latest_successful_call_in_real_time fails it
judge-agreement The last line in the file test_confusion_uses_the_latest_successful_call_in_real_time fails it
judge-agreement The first successful call test_confusion_uses_the_latest_successful_call_in_real_time fails it
judge-agreement The latest call even if it failed test_items_with_no_successful_call_are_counted_and_left_out fails it
judge-agreement A transposed confusion matrix test_confusion_uses_the_latest_successful_call_in_real_time fails it

The solution of the original task also fails the shifted task on the matching fatigue check, and on form, which is computed from both.

A task that a memorized answer cannot pass#

Models have seen the popular benchmarks and the textbook version of most problems. A task tests the agent only if the agent must read it.

training-load-shifted is training-load with one change of substance. The instruction gives time constants of 28 and 5 days, where the common values are 42 and 7. An agent that writes the formula from memory gets all 118 days wrong: fitness on the last day is 77.57 where the instruction gives 84.41.

  • The instruction and the reference solution differ from the original in three lines each. A test checks that.
  • The log is a second one, from another seed. The expected file of the original task cannot pass.
  • The grader names the failure. The solution of the original task fails on test_ctl_uses_the_time_constant_of_the_instruction.

The pair is built to give one number per model: the pass rate on the original minus the pass rate on the shifted task.

tau-bench: an agent that does nothing passes 19 of 50 airline tasks#

tau-bench: tasks passed by an agent that takes no actionMy script loads the benchmark at a pinned commit and calls its own reward code. The published figures are from Zhu et al., "Establishing Best Practices for Building Rigorous Agentic Benchmarks" (2025).
Domain Tasks Passed by an agent that does nothing Published
airline 50 19 (38.0%) 38%
retail 115 7 (6.1%) 6.0%
airline: 19 of 50 tasks
retail: 7 of 115 tasks
passed with no actionnot passed

The script also gives the cause. The reward compares the final database with the database after the reference actions. In all 26 passing tasks the reference actions leave the database as it was: 8 tasks have no reference action, and 18 only read. The 5 read-only tasks that also require an output do not pass.

So a pass on these tasks cannot tell a correct refusal from an agent that did nothing.

SWE-bench Verified: close to half of the score is one repository#

SWE-bench Verified: what the released file holds
Measure Value
Tasks 500, from 12 repositories
Tasks from django/django 231 (46%)
Issue dates, oldest / median / newest 2013-01-25 / 2020-12-27 / 2023-08-07
Tasks where one test checks the fix 345 (69%)

Django supplies 46% of the score. The newest issue is from August 2023, and each issue and its fix are public, so a model trained after that date may have seen them. Zhu et al. (2025) report that an incorrect patch can pass the tests of this benchmark.

Both scripts read pinned public sources and call no model. Each exits with an error if its counts change.

Harbor: what worked and what did not#

Observations from building and running these tasks with Harbor 0.26.0.

Worked

  • The grader runs in its own container. It receives only the files that the task lists under artifacts, so the agent cannot read the expected values.
  • oracle and nop give a two-sided check for the cost of two container runs. One proves that the task can be solved. The other proves that an empty container does not pass.
  • One folder holds the whole task. The instruction, the environment, the reference solution and the grader move together.

Did not

  • harbor run exits with 0 when the reward is 0. A CI step that only runs Harbor passes on a failed task. qa/check_job.py reads each result.json and fails the step.
  • A missing artifacts entry fails silently. The grader finds no output file, and the result looks like a wrong answer.
  • The default grader image installs pytest from PyPI when it builds. That build failed on a network that inspects TLS traffic. These graders use the Python standard library only.
  • harbor lint prints nothing when a task passes. The exit code is the only signal.

Reproduce it#

The commands are the ones in the README.

python -m pip install pytest harbor==0.26.0
python -m pytest -q qa
harbor lint tasks/hrv-from-rr-intervals tasks/training-load tasks/training-load-shifted tasks/judge-agreement
harbor run -p tasks -a oracle -n 4
harbor run -p tasks -a nop -n 4

To run an agent on the tasks, name it and give it a model:

harbor run -p tasks -a hermes -m <provider>/<model>

The two audits are audits/tau_bench.py and audits/swe_bench_verified.py. Both run in CI.

Limits#

  • No agent has run on the tasks. The repository holds no agent results yet, so the pass rate difference between the original and the shifted task is not measured.
  • The audits describe the benchmarks. Neither runs an agent on them.
  • Four tasks are a small set. They come from my own subjects, and every input file is synthetic.
  • The tau-bench rates are a published finding. Zhu et al. (2025) reported them. My script recomputes them from the benchmark's own reward code and reports why each of those tasks passes.
  • The Harbor notes have a narrow base. They cover the task format and the local Docker runner of version 0.26.0, and nothing else.