Evals/Area 2 of 5/Deep Evaluator studies/P3
Deep Evaluator
Which code change caused the behavior change? Experiment code for my method, and three studies on public data.
Headline result
7of21
pairs of systems swap order when "failure" means a specification failure and no longer any failure mode
The result tables- Data
- Public releases of Agent-as-a-Judge (Zhuge et al., 2024) and MAST (Cemri et al., 2025), and a toy harness
- Sample
- 1,638 traces, 366 requirements for each of 3 systems, 5 attribution cases
- Model calls
- None in the tests and the studies
- Checks in CI
- 175 tests, and the three studies on pinned public data
- Repository
- deep-evaluator
The question#
When an agent harness changes (a prompt, a tool, memory, context handling), a benchmark score says whether the result moved. It does not say how the agent's behavior changed or which code change caused it.
Deep Evaluator is my method for those two questions. The paper is in preparation. This page covers the experiment code and three studies on public data that sit beside it.
What I built#
Deep Evaluator compares two versions of an agent harness in two steps.
- Blinded behavioral review. An agentic judge, which is a judge with its own tools, receives the complete traces of both versions on the same task, with version labels removed. It investigates the evidence with its own shell and returns a verdict.
- Attribution. After a directional verdict, the same judge session receives both source trees and states which code change explains what it saw.
Each pair is judged three times in fresh sessions, and a verdict needs a 2-of-3 majority.
| Path | Contents |
|---|---|
experiments/evaluator/ |
The frozen judge, the attribution turn, the schedule builder and the judging driver |
experiments/judge_input.py, experiments/blinding_audit.py |
Score-blind judge inputs, and the audit that checks no version label or source path reaches the judge |
experiments/analysis/agreement.py |
Agreement with a reference: raw agreement, Cohen's kappa, and a seeded task-cluster bootstrap |
experiments/analysis/rep0_assumption_checks.py |
Checks on repeated judgments: order effects, ties, and how far judgments of one pair agree |
experiments/scoring/ |
Native scorers for AppWorld, LongBench v2, SciVisAgentBench and a warning-visibility task set |
experiments/baselines/ |
MT-Bench pairwise judging (FastChat) and Agent-as-a-Judge, run as unchanged pinned upstream code |
experiments/ablations/ |
The judge without tools. The judge without Git is a flag on the frozen judge |
experiments/human_review/ |
A blinded two-reviewer package for collecting human reference labels |
experiments/infra_audit.py, experiments/validate_run.py |
Outcome-blind checks that exclude runs broken by infrastructure faults |
studies/ |
Three small studies on public data, shown under Result |
tests/ |
175 tests that make no model calls |
Design choices#
- The judge sees evidence, never labels. Model inputs are allowlisted copies that exclude scores, diffs, release labels and source paths.
blinding_audit.pychecks the copies. - Behavior first, code second. The judge gives its behavioral verdict before it sees any source code, so the diff cannot bias the verdict.
- Baselines run as their authors wrote them. The FastChat and Agent-as-a-Judge code is pinned by commit and file hash. Tests check the pins when the upstream checkouts are present.
- Infrastructure faults are not results. A run that failed on a transport error or a missing artifact is excluded on both arms, and the reason is recorded.
- Statistics respect the design. Uncertainty comes from resampling tasks, because judgments of one task are not independent.
Result#
The three studies below are first versions. Each one runs from a pinned public source or from files in the repository, calls no model, and runs in CI. None of them is a result of Deep Evaluator: no Deep Evaluator judgment is in the repository yet.
Study 1: an answer key for attribution#
studies/attribution_cases.py builds five pairs of versions of a toy harness. The cause of each behavior change is known, because the script makes the edit.
| Case | What it tests | Passed, base | Passed, candidate | Better | Cause |
|---|---|---|---|---|---|
| regression | One edit, and it breaks tasks | 4 of 7 | 2 of 7 | base | truncate_tool_output |
| improvement | One edit, and it fixes tasks | 4 of 7 | 7 of 7 | candidate | retry_failed_lookup |
| refactor | The code changes and no trace changes | 4 of 7 | 4 of 7 | tie | none |
| cosmetic | The trace text changes and no answer changes | 4 of 7 | 4 of 7 | tie | none |
| decoy | Two edits change the traces, and one causes the failures | 4 of 7 | 2 of 7 | base | truncate_tool_output |
Each case holds both source trees, the traces of both versions with no score in them, and truth.json. The key is checked by reverting: the causal edit alone reproduces the candidate's results, and every other edit alone reproduces the base's results.
The two tie cases are the negative controls. A judge that reports a winner on refactor invents a difference. A judge that blames reword_answer_prefix in decoy attributes the failure to a change that it can see but that causes nothing.
Study 2: a published judge against its human labels#
studies/judge_against_humans.py reads the judgments that the Agent-as-a-Judge authors released (Zhuge et al., 2024) and recomputes how often their judge agrees with the human judges. The three published alignment rates reproduce from the released files, to rounding.
| System judged | Humans say satisfied | Agent judge, paper | Agent judge, recomputed | Cohen's kappa | A judge that always says no | LLM judge, paper | Agreement with the human judges, 0 to 100% |
|---|---|---|---|---|---|---|---|
| MetaGPT | 22.1% | 92.07% | 92.08% | 0.78 | 77.87% | 68.86% | |
| GPT-Pilot | 44.5% | 86.61% | 86.61% | 0.73 | 55.46% | 71.85% | |
| OpenHands | 42.9% | 90.16% | 90.16% | 0.80 | 57.10% | 70.76% |
- An alignment rate needs its baseline. On MetaGPT the humans mark 22.1% of the requirements as satisfied, so a judge that always answers "not satisfied" scores 77.87%. The paper's LLM judge scores 68.86% there, below that baseline.
- The agent judge holds up after the correction for chance. Its kappa is 0.73 to 0.80 on the three systems.
- Its errors lean one way. On all three systems it says "satisfied" where the humans say no more often than the reverse.
| System judged | Judge says satisfied, humans say no | Judge says not satisfied, humans say yes |
|---|---|---|
| MetaGPT | 17 | 12 |
| GPT-Pilot | 28 | 21 |
| OpenHands | 19 | 17 |
Study 3: what the granularity of a failure taxonomy does to the verdict#
studies/failure_taxonomy.py uses the public MAST traces (Cemri et al., 2025). The definition of "failure" decides which system looks worse.
| A trace fails when it has | Traces that fail | Share of the 1,638 traces |
|---|---|---|
| any failure mode | 75.5% | |
| a specification failure (1.x) | 58.2% | |
| an inter-agent failure (2.x) | 53.5% | |
| a verification failure (3.x) | 55.4% |
ChatDev and MetaGPT fail on 94% and 95% of their traces under "any failure mode". Under "a specification failure" ChatDev fails on 93% and MetaGPT on 57%. Under "a verification failure" the order reverses: 50% against 86%.
| Row | System | Traces | Any failure mode | A specification failure (1.x) | An inter-agent failure (2.x) | A verification failure (3.x) | Failure modes per trace |
|---|---|---|---|---|---|---|---|
| 1 | AG2 | 597 | 1.55 | ||||
| 2 | AppWorld | 30 | 3.67 | ||||
| 3 | ChatDev | 326 | 4.66 | ||||
| 4 | HyperAgent | 30 | 6.17 | ||||
| 5 | Magentic | 195 | 4.08 | ||||
| 6 | MetaGPT | 430 | 3.42 | ||||
| 7 | OpenManus | 30 | 4.33 |
Of the 21 pairs of systems, 7 swap order between "any failure mode" and "a specification failure".
| Definition, against "any failure mode" | Pairs of systems | Pairs that swap order | One square per pair |
|---|---|---|---|
| a specification failure (1.x) | 21 | 7 | |
| an inter-agent failure (2.x) | 21 | 4 | |
| a verification failure (3.x) | 21 | 5 |
The failure modes also overlap across the category lines. A trace has 3.14 failure modes on average, and one example of a pair that occurs together is "Loss of Conversation History" (1.4) with "Conversation Reset" (2.1).
| Failure modes | Overlap (Jaccard) |
|---|---|
| 1.3 and 1.5 | 0.60 |
| 2.6 and 3.3 | 0.51 |
| 1.4 and 2.1 | 0.46 |
| 2.6 and 3.2 | 0.45 |
| 1.1 and 2.3 | 0.44 |
Reproduce it#
The commands are the ones in the README. The tests make no model calls.
python -m pip install -e . pytest
python -m pytest -qThe three studies:
python -m studies.attribution_cases # writes studies/cases/
python -m studies.judge_against_humans # clones the pinned judgments, exits 1 on a mismatch with the paper
python -m studies.failure_taxonomy --full # downloads 200 MB of traces, checks the SHA-256Six tests skip without the pinned FastChat and Agent-as-a-Judge checkouts, an archived schedule, or matplotlib.
Limits#
- No Deep Evaluator result is shown here. The repository holds the code. The study protocol, the results and the benchmark-derived inputs are released with the paper, which is in preparation.
- No judge has run on the five attribution cases. The answer key exists. I have not yet measured a judge against it.
- The toy harness is scripted. Every run is deterministic, and each version has 7 tasks. The cases test whether a judge finds a known cause. They do not show how a judge behaves on a real harness.
- The kappa of the paper's LLM judge cannot be recomputed. The released files hold the agent judge's verdicts. They do not hold the LLM judge's verdicts.
- The failure rates are not a ranking. The systems ran different benchmarks, so the per-system rates compare rows of this dataset.
- The failure labels come from the authors' LLM annotator. The human labels in the release are too few to settle the granularity question: the file has 19 traces labeled by three annotators.
- One taxonomy. A second taxonomy on the same traces is open. That needs a judge run.
- No benchmark data is in the repository. The AppWorld authors ask that their tasks and API documentation stay out of plain text online, so every file derived from them is left out.