Sara Ghorayeb

Theme 1 of 3 / Evaluation of AI agents

Evals

I build evaluation infrastructure for AI agents, and I measure whether their judges agree with people.

I lead agent evaluation at GSK, where I built the evaluation infrastructure behind every release and pull request of a biomedical research agent that 3,000 scientists use. My first-author research is on agentic judges, judge calibration and causal failure analysis.

  1. Judge reliabilityJudge calibration, Verdict aggregation
  2. Deep Evaluator studiesDeep Evaluator
  3. BenchmarksHarbor tasks and benchmark audits
  4. Eval infrastructureEval pipeline
  5. Hands-on LLM workFew-shot designs
Simulation

Made-up judges vote on made-up traces in your browser. None of these numbers is a result of mine. My results are in the panels below.

Agreement between the simulated judges, Fleiss' kappa

0.00

95% interval

0 traces

0%of majority verdicts match the true label

Agreement as traces arrive

line: estimate / band: 95% bootstrap interval / dashed: value the settings imply

5 votes on each trace
15% chance of a wrong vote
0% judges make the same mistake

Try it. Raise shared mistakes. The judges agree more, and their majority is wrong more often.

Projects

Six repositories in five areas. Each entry shows one result with its data and its sample size. The linked page has the method, the result tables, the commands and the limits.

P1judge-calibrationArea 1 of 5 / Judge reliability

Judge calibration

Does an LLM judge agree with human labels?

A protocol that another person can run again, on the MT-Bench human judgments. It first reproduces the 8 cells of Table 5 of Zheng et al. (2023), then measures a current judge against the majority label of the expert labelers.

0.58

Cohen's kappa of the judge against the human majority label, 95% interval 0.46 to 0.71

121 MT-Bench items, the same items for both judges. Dot: kappa. Bar: 95% interval from 2,000 bootstrap replicates that resample questions. The two intervals overlap, so this sample does not establish a difference between the two judges.
Agreement, per-class precision and recall, drift zones
P2verdict-aggregationArea 1 of 5 / Judge reliability

Verdict aggregation

How should several verdicts become one?

Majority vote against the Dawid-Skene model and an item response model, on four public crowdsourcing datasets. The 12 published values for the first two methods reproduce. A simulation then tests judges that share their mistakes.

1,184of1,517

gold items of one dataset have no vote in the released files, and the authors' scoring code gives each of them a random label

333 with votes1,184 with a random label

Majority vote accuracy, percent

All 1,517 gold items36.19

The 333 that have votes75.98

Majority vote accuracy in percent on S_Adult, from Zheng et al. (VLDB 2017). This is a statement about the released archive. I have not asked the authors if their own copy of the data differs.
Reproduction, votes per item, shared errors, run-to-run noise
P3deep-evaluatorArea 2 of 5 / Deep Evaluator studies

Deep Evaluator

Which code change caused the behavior change?

Experiment code for my method. An agentic judge compares two harness versions on complete traces with the version labels removed, then attributes the difference to a code change. Three studies on public data sit beside the code.

7of21

pairs of systems swap order when "failure" means a specification failure and no longer any failure mode

Attribution answer key, a published judge recomputed, failure taxonomy

Share of traces that fail, under three definitions of failure

ChatDevMetaGPT

Share of traces that fail under three definitions of failure, for two of the seven systems. 1,638 public MAST traces (Cemri et al., 2025), labeled by the authors' LLM annotator. The systems ran different benchmarks, so this is not a ranking. The study is separate from the Deep Evaluator results, which are released with the paper.
P4harbor-tasksArea 3 of 5 / Benchmarks

Harbor tasks and benchmark audits

Is the task or the grader at fault?

Four terminal tasks in the Harbor format, with graders tested against 14 plausible wrong solutions. Audits of tau-bench and SWE-bench Verified, with a limit of each. Notes on what worked in Harbor and what did not.

19of50

tau-bench airline tasks are passed by an agent that takes no action

Airline domain19 of 50

Retail domain7 of 115 (6.1%)

passed with no actionnot passed

My script calls the benchmark's own reward code at a pinned commit. Zhu et al. (2025) published 38% and 6.0% for the two domains. On these tasks a pass cannot tell a correct refusal from an agent that did nothing. The audit describes the benchmark and runs no agent on it.
Grader checks, a task that a memorized answer fails, two audits
P5eval-pipelineArea 4 of 5 / Eval infrastructure

Eval pipeline

What happens on every commit?

A watcher, a job queue, a worker, an API, a command line and a dashboard, written with the Python standard library. Each new commit of an agent is evaluated against the commit before it. An infrastructure fault is retried and never becomes a verdict.

5of7

demo tasks regress at the commit that cuts tool output to 80 characters, and the pipeline marks that commit

Task8479a28e39b953marked0c9919af99c749
bike/Gearspasspassfailfail
bike/Tyre pressure in barpasspasspasspass
plan/Weeksfailpassfailfail
pool/Lane countpasspasspasspass
pool/Length in metrespasspassfailfail
track/Lanesfailpassfailfail
track/Lap in metresfailpassfailfail
Passed4 / 77 / 72 / 72 / 7

the task changed at this commitsame as the commit before

A scripted lookup agent with 7 tasks and 4 commits, each with a known effect, so the pipeline has an answer key. A further job names a commit that does not exist. It fails after 3 attempts and stores no verdict. The eval is a pass or fail script, and no agentic judge is connected.
The demo with its answer key, the dashboard, design decisions
P6few-shotArea 5 of 5 / Hands-on LLM work

Few-shot designs

When does a few-shot prompt earn its cost?

Free-text training notes become records with four fields. I built the labeled data, a rule baseline, the prompt, four ways to choose the examples and a strict scorer. No model has run on it yet.

The baseline, the two notes it gets wrong, example selection

22of24

notes are parsed correctly by 30 lines of keyword rules, so that is the bar for the prompt

Keyword rules, 30 lines22 of 24

Few-shot promptthe model run is open

whole record rightwrong: both notes hold two numbers of one kindnot run

24 eval notes, written and labeled for this project. The model run is open.

The two other themes