Sara Ghorayeb

Evals/Area 4 of 5/Eval infrastructure/P5

Eval pipeline

What happens on every commit? A watcher, a job queue, a worker, an API and a dashboard that compare each commit with the one before it.

Headline result

5of7

demo tasks regress at the commit that cuts tool output to 80 characters, and the pipeline marks that commit

The result tables
Data
A demo repository with a scripted lookup agent
Sample
7 tasks, 4 commits, 4 jobs with a known outcome
Model calls
None
Checks in CI
26 tests, the demo in one process, the demo with each eval in a container, the compose stack
Repository
eval-pipeline

The question#

What did this commit change in the agent's behavior? To answer that on every commit, a pipeline has to compare each commit with the one before it on the same tasks. It also has to keep a run that broke for a reason outside the agent from being read as a regression.

What I built#

A small pipeline that evaluates each new commit of an agent against the commit before it, and shows which tasks changed. It uses the Python standard library only, and no step calls a model.

  1. Git repositorythe agent under test
  2. Watcherqueues one comparison per new commit
  3. Job queueSQLite. It also takes jobs from evalpipe submit over HTTP
  4. Workerexports both commits, runs the eval on each, compares task by task
  5. API and dashboardjob state as JSON, and every task across the commits

Design decisions#

  • An infrastructure fault is never a verdict. A timeout, a crash, output that is not JSON, or a commit that cannot be read sends the job back to the queue. After three attempts the job is failed and has no result. It is never counted as a regression.
  • The eval command is not in the repository under test. It is in the targets file. A commit cannot change how it is graded.
  • Each commit runs from a clean export. git archive writes the commit into an empty directory with no .git folder. The directory is removed after the job.
  • Container isolation is one flag. With --isolation docker the eval runs with no network and a read-only source tree.
  • The API accepts only known targets and commit hashes. A branch name, an option or a shell fragment is rejected with status 400 before anything is queued.
  • A job is claimed in one SQL statement, so two workers cannot run the same job. The same comparison submitted twice is one job.
  • A job whose worker stopped returns to the queue after a time limit.
Layout of the code
File Content
evalpipe/store.py The job table: submit, claim, finish, retry, requeue
evalpipe/runner.py Export a commit, run the eval, compare two results
evalpipe/service.py The watcher step and the worker step
evalpipe/api.py The HTTP API and the dashboard
evalpipe/cli.py The command line
evalpipe/demo_target.py The demo repository
tests/test_pipeline.py 26 tests. None needs Docker

Result#

The pipeline gives the known outcome on all four demo jobs#

evalpipe demo builds a Git repository with a scripted lookup agent, seven tasks and four commits. Each commit has a known effect, so the pipeline has an answer key.

The demo target: four jobs with a known outcomeA scripted lookup agent with seven tasks. The demo exits with an error if any row differs.
Commit Change Passed before Passed after Verdict
e39b953 Retry a lookup that timed out 4 of 7 7 of 7 candidate better, 3 tasks fixed
0c9919a Cut tool output to 80 characters 7 of 7 2 of 7 base better, 5 tasks regressed
f99c749 Rename a variable 2 of 7 2 of 7 no change
0000000 A commit that does not exist no result no result failed after 3 attempts, no verdict
task passedtask failed

The last row is the rule about infrastructure faults. The job names a commit that does not exist. The pipeline retries the job, then marks it failed and stores no verdict.

Every task across the commits#

The dashboard marks the commit where a task changed. Five tasks turn from pass to fail at 0c9919a, the commit that cuts tool output to 80 characters.

Task history of the demo target. Each column is a commit.
Task8479a28e39b9530c9919af99c749
bike/Gearspasspassfailfail
bike/Tyre pressure in barpasspasspasspass
plan/Weeksfailpassfailfail
pool/Lane countpasspasspasspass
pool/Length in metrespasspassfailfail
track/Lanesfailpassfailfail
track/Lap in metresfailpassfailfail
Passed4 / 77 / 72 / 72 / 7
the task changed at this commitsame as the commit before

The dashboard#

Screenshot of the dashboard. Six counters: 3 comparisons done, 1 candidate better, 1 base better or mixed, 1 no change, 0 queued or running, 1 failed with no verdict. Under them the task history grid and a table of the four comparisons with status, attempts, verdict, the tasks that changed and the last error.
A screenshot of the dashboard after evalpipe demo, from the repository. The failed job shows its 3 attempts and its error, and its verdict cell is empty.

CI runs the demo three ways: in one process, with each eval in a container, and as the compose stack of three containers.

Reproduce it#

The commands are the ones in the README.

python -m pip install -e ".[test]"
python -m pytest -q

evalpipe demo --dir data                        # the whole pipeline in one process; writes data/dashboard.html
evalpipe demo --dir data --isolation docker     # the same, with each eval in a container

docker compose up -d --build                    # API, watcher and worker as three containers
evalpipe status                                 # reads http://127.0.0.1:8080/api/jobs
evalpipe submit --target demo --base <commit> --candidate <commit>

A target is one entry in targets.json. The command must print a JSON object of task name to true or false.

{"demo": {"repo": "/data/targets/demo", "branch": "main", "command": ["python3", "run_eval.py"], "image": "python:3.12-slim", "timeout": 60}}

Limits#

  • The eval is a pass or fail script. No agentic judge is connected. runner.run_arm is the one place where a judge would go.
  • The demo agent is scripted, so every run gives the same result. A real agent needs repeated runs per commit.
  • The API has no authentication. The compose file binds it to 127.0.0.1.
  • It has run on a laptop and in CI. It is not deployed on a server.