Evals/Area 4 of 5/Eval infrastructure/P5
Eval pipeline
What happens on every commit? A watcher, a job queue, a worker, an API and a dashboard that compare each commit with the one before it.
Headline result
5of7
demo tasks regress at the commit that cuts tool output to 80 characters, and the pipeline marks that commit
The result tables- Data
- A demo repository with a scripted lookup agent
- Sample
- 7 tasks, 4 commits, 4 jobs with a known outcome
- Model calls
- None
- Checks in CI
- 26 tests, the demo in one process, the demo with each eval in a container, the compose stack
- Repository
- eval-pipeline
The question#
What did this commit change in the agent's behavior? To answer that on every commit, a pipeline has to compare each commit with the one before it on the same tasks. It also has to keep a run that broke for a reason outside the agent from being read as a regression.
What I built#
A small pipeline that evaluates each new commit of an agent against the commit before it, and shows which tasks changed. It uses the Python standard library only, and no step calls a model.
- Git repositorythe agent under test
- Watcherqueues one comparison per new commit
- Job queueSQLite. It also takes jobs from
evalpipe submitover HTTP - Workerexports both commits, runs the eval on each, compares task by task
- API and dashboardjob state as JSON, and every task across the commits
Design decisions#
- An infrastructure fault is never a verdict. A timeout, a crash, output that is not JSON, or a commit that cannot be read sends the job back to the queue. After three attempts the job is
failedand has no result. It is never counted as a regression. - The eval command is not in the repository under test. It is in the targets file. A commit cannot change how it is graded.
- Each commit runs from a clean export.
git archivewrites the commit into an empty directory with no.gitfolder. The directory is removed after the job. - Container isolation is one flag. With
--isolation dockerthe eval runs with no network and a read-only source tree. - The API accepts only known targets and commit hashes. A branch name, an option or a shell fragment is rejected with status 400 before anything is queued.
- A job is claimed in one SQL statement, so two workers cannot run the same job. The same comparison submitted twice is one job.
- A job whose worker stopped returns to the queue after a time limit.
| File | Content |
|---|---|
evalpipe/store.py |
The job table: submit, claim, finish, retry, requeue |
evalpipe/runner.py |
Export a commit, run the eval, compare two results |
evalpipe/service.py |
The watcher step and the worker step |
evalpipe/api.py |
The HTTP API and the dashboard |
evalpipe/cli.py |
The command line |
evalpipe/demo_target.py |
The demo repository |
tests/test_pipeline.py |
26 tests. None needs Docker |
Result#
The pipeline gives the known outcome on all four demo jobs#
evalpipe demo builds a Git repository with a scripted lookup agent, seven tasks and four commits. Each commit has a known effect, so the pipeline has an answer key.
| Commit | Change | Passed before | Passed after | Verdict |
|---|---|---|---|---|
e39b953 |
Retry a lookup that timed out | 4 of 7 | 7 of 7 | candidate better, 3 tasks fixed |
0c9919a |
Cut tool output to 80 characters | 7 of 7 | 2 of 7 | base better, 5 tasks regressed |
f99c749 |
Rename a variable | 2 of 7 | 2 of 7 | no change |
0000000 |
A commit that does not exist | no result | no result | failed after 3 attempts, no verdict |
The last row is the rule about infrastructure faults. The job names a commit that does not exist. The pipeline retries the job, then marks it failed and stores no verdict.
Every task across the commits#
The dashboard marks the commit where a task changed. Five tasks turn from pass to fail at 0c9919a, the commit that cuts tool output to 80 characters.
| Task | 8479a28 | e39b953 | 0c9919a | f99c749 |
|---|---|---|---|---|
| bike/Gears | pass | pass | fail | fail |
| bike/Tyre pressure in bar | pass | pass | pass | pass |
| plan/Weeks | fail | pass | fail | fail |
| pool/Lane count | pass | pass | pass | pass |
| pool/Length in metres | pass | pass | fail | fail |
| track/Lanes | fail | pass | fail | fail |
| track/Lap in metres | fail | pass | fail | fail |
| Passed | 4 / 7 | 7 / 7 | 2 / 7 | 2 / 7 |
The dashboard#
evalpipe demo, from the repository. The failed job shows its 3 attempts and its error, and its verdict cell is empty.CI runs the demo three ways: in one process, with each eval in a container, and as the compose stack of three containers.
Reproduce it#
The commands are the ones in the README.
python -m pip install -e ".[test]"
python -m pytest -q
evalpipe demo --dir data # the whole pipeline in one process; writes data/dashboard.html
evalpipe demo --dir data --isolation docker # the same, with each eval in a container
docker compose up -d --build # API, watcher and worker as three containers
evalpipe status # reads http://127.0.0.1:8080/api/jobs
evalpipe submit --target demo --base <commit> --candidate <commit>A target is one entry in targets.json. The command must print a JSON object of task name to true or false.
{"demo": {"repo": "/data/targets/demo", "branch": "main", "command": ["python3", "run_eval.py"], "image": "python:3.12-slim", "timeout": 60}}Limits#
- The eval is a pass or fail script. No agentic judge is connected.
runner.run_armis the one place where a judge would go. - The demo agent is scripted, so every run gives the same result. A real agent needs repeated runs per commit.
- The API has no authentication. The compose file binds it to
127.0.0.1. - It has run on a laptop and in CI. It is not deployed on a server.