I build evaluation infrastructure for AI agents, and I measure whether their judges agree with people.
I lead agent evaluation at GSK, where I built the evaluation infrastructure behind every release and pull request of a biomedical research agent that 3,000 scientists use. My first-author research is on agentic judges, judge calibration and causal failure analysis.
Made-up judges vote on made-up traces in your browser. None of these numbers is a result of mine. My results are in the panels below.
pass votefail votemajority wrong
Agreement between the simulated judges, Fleiss' kappa
0.00
95% interval
0 traces
0%of majority verdicts match the true label
Agreement as traces arrive
line: estimate / band: 95% bootstrap interval / dashed: value the settings imply
votes on each trace
chance of a wrong vote
judges make the same mistake
Try it. Raise shared mistakes. The judges agree more, and their majority is wrong more often.
01Projects
Six repositories in five areas. Each entry shows one result with its data and its sample size. The linked page has the method, the result tables, the commands and the limits.
P1judge-calibrationArea 1 of 5 / Judge reliability
A protocol that another person can run again, on the MT-Bench human judgments. It first reproduces the 8 cells of Table 5 of Zheng et al. (2023), then measures a current judge against the majority label of the expert labelers.
0.58
Cohen's kappa of the judge against the human majority label, 95% interval 0.46 to 0.71
overlap
gpt-6.1-sol0.58 [0.46, 0.71]
GPT-4, 20230.55 [0.43, 0.66]
00.51
121 MT-Bench items, the same items for both judges. Dot: kappa. Bar: 95% interval from 2,000 bootstrap replicates that resample questions. The two intervals overlap, so this sample does not establish a difference between the two judges.Agreement, per-class precision and recall, drift zones
P2verdict-aggregationArea 1 of 5 / Judge reliability
Majority vote against the Dawid-Skene model and an item response model, on four public crowdsourcing datasets. The 12 published values for the first two methods reproduce. A simulation then tests judges that share their mistakes.
1,184of1,517
gold items of one dataset have no vote in the released files, and the authors' scoring code gives each of them a random label
333 with votes1,184 with a random label
Majority vote accuracy, percent
All 1,517 gold items36.19
The 333 that have votes75.98
Majority vote accuracy in percent on S_Adult, from Zheng et al. (VLDB 2017). This is a statement about the released archive. I have not asked the authors if their own copy of the data differs.Reproduction, votes per item, shared errors, run-to-run noise
P3deep-evaluatorArea 2 of 5 / Deep Evaluator studies
Experiment code for my method. An agentic judge compares two harness versions on complete traces with the version labels removed, then attributes the difference to a code change. Three studies on public data sit beside the code.
7of21
pairs of systems swap order when "failure" means a specification failure and no longer any failure mode
Share of traces that fail, under three definitions of failure
100%75%50%95%57%86%94%93%50%
Any failure modeA specification failureA verification failure
ChatDevMetaGPT
Share of traces that fail under three definitions of failure, for two of the seven systems. 1,638 public MAST traces (Cemri et al., 2025), labeled by the authors' LLM annotator. The systems ran different benchmarks, so this is not a ranking. The study is separate from the Deep Evaluator results, which are released with the paper.
Four terminal tasks in the Harbor format, with graders tested against 14 plausible wrong solutions. Audits of tau-bench and SWE-bench Verified, with a limit of each. Notes on what worked in Harbor and what did not.
19of50
tau-bench airline tasks are passed by an agent that takes no action
Airline domain19 of 50
Retail domain7 of 115 (6.1%)
passed with no actionnot passed
My script calls the benchmark's own reward code at a pinned commit. Zhu et al. (2025) published 38% and 6.0% for the two domains. On these tasks a pass cannot tell a correct refusal from an agent that did nothing. The audit describes the benchmark and runs no agent on it.Grader checks, a task that a memorized answer fails, two audits
A watcher, a job queue, a worker, an API, a command line and a dashboard, written with the Python standard library. Each new commit of an agent is evaluated against the commit before it. An infrastructure fault is retried and never becomes a verdict.
5of7
demo tasks regress at the commit that cuts tool output to 80 characters, and the pipeline marks that commit
Task
8479a28
e39b953
marked0c9919a
f99c749
bike/Gears
pass
pass
fail
fail
bike/Tyre pressure in bar
pass
pass
pass
pass
plan/Weeks
fail
pass
fail
fail
pool/Lane count
pass
pass
pass
pass
pool/Length in metres
pass
pass
fail
fail
track/Lanes
fail
pass
fail
fail
track/Lap in metres
fail
pass
fail
fail
Passed
4 / 7
7 / 7
2 / 7
2 / 7
the task changed at this commitsame as the commit before
A scripted lookup agent with 7 tasks and 4 commits, each with a known effect, so the pipeline has an answer key. A further job names a commit that does not exist. It fails after 3 attempts and stores no verdict. The eval is a pass or fail script, and no agentic judge is connected.The demo with its answer key, the dashboard, design decisions
Free-text training notes become records with four fields. I built the labeled data, a rule baseline, the prompt, four ways to choose the examples and a strict scorer. No model has run on it yet.