Sara Ghorayeb

Evals/Area 2 of 5/Deep Evaluator studies/P3

Deep Evaluator

Which code change caused the behavior change? Experiment code for my method, and three studies on public data.

Headline result

7of21

pairs of systems swap order when "failure" means a specification failure and no longer any failure mode

The result tables
Data
Public releases of Agent-as-a-Judge (Zhuge et al., 2024) and MAST (Cemri et al., 2025), and a toy harness
Sample
1,638 traces, 366 requirements for each of 3 systems, 5 attribution cases
Model calls
None in the tests and the studies
Checks in CI
175 tests, and the three studies on pinned public data
Repository
deep-evaluator

The question#

When an agent harness changes (a prompt, a tool, memory, context handling), a benchmark score says whether the result moved. It does not say how the agent's behavior changed or which code change caused it.

Deep Evaluator is my method for those two questions. The paper is in preparation. This page covers the experiment code and three studies on public data that sit beside it.

What I built#

Deep Evaluator compares two versions of an agent harness in two steps.

  1. Blinded behavioral review. An agentic judge, which is a judge with its own tools, receives the complete traces of both versions on the same task, with version labels removed. It investigates the evidence with its own shell and returns a verdict.
  2. Attribution. After a directional verdict, the same judge session receives both source trees and states which code change explains what it saw.

Each pair is judged three times in fresh sessions, and a verdict needs a 2-of-3 majority.

What the repository holds
Path Contents
experiments/evaluator/ The frozen judge, the attribution turn, the schedule builder and the judging driver
experiments/judge_input.py, experiments/blinding_audit.py Score-blind judge inputs, and the audit that checks no version label or source path reaches the judge
experiments/analysis/agreement.py Agreement with a reference: raw agreement, Cohen's kappa, and a seeded task-cluster bootstrap
experiments/analysis/rep0_assumption_checks.py Checks on repeated judgments: order effects, ties, and how far judgments of one pair agree
experiments/scoring/ Native scorers for AppWorld, LongBench v2, SciVisAgentBench and a warning-visibility task set
experiments/baselines/ MT-Bench pairwise judging (FastChat) and Agent-as-a-Judge, run as unchanged pinned upstream code
experiments/ablations/ The judge without tools. The judge without Git is a flag on the frozen judge
experiments/human_review/ A blinded two-reviewer package for collecting human reference labels
experiments/infra_audit.py, experiments/validate_run.py Outcome-blind checks that exclude runs broken by infrastructure faults
studies/ Three small studies on public data, shown under Result
tests/ 175 tests that make no model calls

Design choices#

  • The judge sees evidence, never labels. Model inputs are allowlisted copies that exclude scores, diffs, release labels and source paths. blinding_audit.py checks the copies.
  • Behavior first, code second. The judge gives its behavioral verdict before it sees any source code, so the diff cannot bias the verdict.
  • Baselines run as their authors wrote them. The FastChat and Agent-as-a-Judge code is pinned by commit and file hash. Tests check the pins when the upstream checkouts are present.
  • Infrastructure faults are not results. A run that failed on a transport error or a missing artifact is excluded on both arms, and the reason is recorded.
  • Statistics respect the design. Uncertainty comes from resampling tasks, because judgments of one task are not independent.

Result#

The three studies below are first versions. Each one runs from a pinned public source or from files in the repository, calls no model, and runs in CI. None of them is a result of Deep Evaluator: no Deep Evaluator judgment is in the repository yet.

Study 1: an answer key for attribution#

studies/attribution_cases.py builds five pairs of versions of a toy harness. The cause of each behavior change is known, because the script makes the edit.

Five pairs of versions of a toy harness, with the cause of each behavior change known7 tasks per version. The script makes each edit, so the answer key is exact.
Case What it tests Passed, base Passed, candidate Better Cause
regression One edit, and it breaks tasks 4 of 7 2 of 7 base truncate_tool_output
improvement One edit, and it fixes tasks 4 of 7 7 of 7 candidate retry_failed_lookup
refactor The code changes and no trace changes 4 of 7 4 of 7 tie none
cosmetic The trace text changes and no answer changes 4 of 7 4 of 7 tie none
decoy Two edits change the traces, and one causes the failures 4 of 7 2 of 7 base truncate_tool_output
task passedtask failed

Each case holds both source trees, the traces of both versions with no score in them, and truth.json. The key is checked by reverting: the causal edit alone reproduces the candidate's results, and every other edit alone reproduces the base's results.

The two tie cases are the negative controls. A judge that reports a winner on refactor invents a difference. A judge that blames reword_answer_prefix in decoy attributes the failure to a change that it can see but that causes nothing.

Study 2: a published judge against its human labels#

studies/judge_against_humans.py reads the judgments that the Agent-as-a-Judge authors released (Zhuge et al., 2024) and recomputes how often their judge agrees with the human judges. The three published alignment rates reproduce from the released files, to rounding.

Agent-as-a-Judge against its human judges, recomputed from the released judgments366 requirements per system. Zhuge et al. (2024). The last four columns are not in the paper's table.
System judged Humans say satisfied Agent judge, paper Agent judge, recomputed Cohen's kappa A judge that always says no LLM judge, paper Agreement with the human judges, 0 to 100%
MetaGPT 22.1% 92.07% 92.08% 0.78 77.87% 68.86%
GPT-Pilot 44.5% 86.61% 86.61% 0.73 55.46% 71.85%
OpenHands 42.9% 90.16% 90.16% 0.80 57.10% 70.76%
a judge that always says "not satisfied"LLM judge, as the paper reports itagent judge, recomputed
  • An alignment rate needs its baseline. On MetaGPT the humans mark 22.1% of the requirements as satisfied, so a judge that always answers "not satisfied" scores 77.87%. The paper's LLM judge scores 68.86% there, below that baseline.
  • The agent judge holds up after the correction for chance. Its kappa is 0.73 to 0.80 on the three systems.
  • Its errors lean one way. On all three systems it says "satisfied" where the humans say no more often than the reverse.
The two kinds of disagreement of the agent judgeCounts of requirements, of 366 per system.
System judged Judge says satisfied, humans say no Judge says not satisfied, humans say yes
MetaGPT 17 12
GPT-Pilot 28 21
OpenHands 19 17

Study 3: what the granularity of a failure taxonomy does to the verdict#

studies/failure_taxonomy.py uses the public MAST traces (Cemri et al., 2025). The definition of "failure" decides which system looks worse.

The same traces under four definitions of failure1,638 MAST traces of seven multi-agent systems, each labeled with 14 failure modes in 3 categories by the authors' LLM annotator.
A trace fails when it has Traces that fail Share of the 1,638 traces
any failure mode 75.5%
a specification failure (1.x) 58.2%
an inter-agent failure (2.x) 53.5%
a verification failure (3.x) 55.4%

ChatDev and MetaGPT fail on 94% and 95% of their traces under "any failure mode". Under "a specification failure" ChatDev fails on 93% and MetaGPT on 57%. Under "a verification failure" the order reverses: 50% against 86%.

Failure rate per system under each definitionShare of each system's traces that fail. Sort by a definition to see the order of the systems change.
Row System Traces Any failure mode A specification failure (1.x) An inter-agent failure (2.x) A verification failure (3.x) Failure modes per trace
1 AG2 597 48% 36% 36% 30% 1.55
2 AppWorld 30 97% 47% 77% 87% 3.67
3 ChatDev 326 94% 93% 63% 50% 4.66
4 HyperAgent 30 90% 83% 80% 73% 6.17
5 Magentic 195 78% 64% 60% 66% 4.08
6 MetaGPT 430 95% 57% 63% 86% 3.42
7 OpenManus 30 97% 97% 77% 57% 4.33

Of the 21 pairs of systems, 7 swap order between "any failure mode" and "a specification failure".

Pairs of systems that swap order when the definition changes
Definition, against "any failure mode" Pairs of systems Pairs that swap order One square per pair
a specification failure (1.x) 21 7
an inter-agent failure (2.x) 21 4
a verification failure (3.x) 21 5
pair that swaps orderpair that keeps its order

The failure modes also overlap across the category lines. A trace has 3.14 failure modes on average, and one example of a pair that occurs together is "Loss of Conversation History" (1.4) with "Conversation Reset" (2.1).

The five pairs of failure modes that most often occur togetherThe number before the point is the category. Four of the five pairs sit in two different categories.
Failure modes Overlap (Jaccard)
1.3 and 1.5 0.60
2.6 and 3.3 0.51
1.4 and 2.1 0.46
2.6 and 3.2 0.45
1.1 and 2.3 0.44

Reproduce it#

The commands are the ones in the README. The tests make no model calls.

python -m pip install -e . pytest
python -m pytest -q

The three studies:

python -m studies.attribution_cases                 # writes studies/cases/
python -m studies.judge_against_humans              # clones the pinned judgments, exits 1 on a mismatch with the paper
python -m studies.failure_taxonomy --full           # downloads 200 MB of traces, checks the SHA-256

Six tests skip without the pinned FastChat and Agent-as-a-Judge checkouts, an archived schedule, or matplotlib.

Limits#

  • No Deep Evaluator result is shown here. The repository holds the code. The study protocol, the results and the benchmark-derived inputs are released with the paper, which is in preparation.
  • No judge has run on the five attribution cases. The answer key exists. I have not yet measured a judge against it.
  • The toy harness is scripted. Every run is deterministic, and each version has 7 tasks. The cases test whether a judge finds a known cause. They do not show how a judge behaves on a real harness.
  • The kappa of the paper's LLM judge cannot be recomputed. The released files hold the agent judge's verdicts. They do not hold the LLM judge's verdicts.
  • The failure rates are not a ranking. The systems ran different benchmarks, so the per-system rates compare rows of this dataset.
  • The failure labels come from the authors' LLM annotator. The human labels in the release are too few to settle the granularity question: the file has 19 traces labeled by three annotators.
  • One taxonomy. A second taxonomy on the same traces is open. That needs a judge run.
  • No benchmark data is in the repository. The AppWorld authors ask that their tasks and API documentation stay out of plain text online, so every file derived from them is left out.