Sara Ghorayeb

Evals/Area 1 of 5/Judge reliability/P1

Judge calibration

How far can I trust an LLM judge? A calibration protocol against human labels, written so that another person can run it again.

Headline result

0.58

Cohen's kappa of the judge against the human majority label, 95% interval 0.46 to 0.71

The result tables
Data
MT-Bench human judgments, Zheng et al. (2023)
Sample
121 items with a verdict in both answer orders, of 160 sampled
Model calls
243 of 320, then the run stopped
Checks in CI
Tests, and the reproduction of the published table from the pinned public data
Repository
judge-calibration
MIT license

The question#

How far can I trust this judge? The protocol answers that one question. It reports agreement with human labels above the chance level, precision and recall per class, the segments where the judge is weaker, and how often the verdict changes with the order of the answers.

The example calibrates gpt-6.1-sol as a pairwise judge on MT-Bench. The reference is the expert human votes that Zheng et al. released in 2023.

What I built#

A command line tool with five steps. Four of them make no model call.

  1. fetch downloads the human votes, the questions and the unchanged FastChat judge prompts. Each file is pinned by commit and SHA-256.
  2. reproduce recomputes Table 5 of Zheng et al. (2023) from the released votes. A calibration on data that the code reads wrongly is worthless, so this step runs in CI.
  3. sample draws a seeded sample of items that have a human majority label, with the same number per category and turn.
  4. judge shows each item to the judge twice, once per answer order. A win counts only when the judge names the same model in both orders. This is FastChat's rule.
  5. report writes the statistics to report.md and report.json.

How the judge is called#

judge calls the model through Hermes Agent with the openai-codex provider. Each call is a fresh session with user configuration, memory and rules off. ask_hermes in judge_calibration/judge.py is the one function to replace for another backend.

The command appends each result at once, stops after five failed calls in a row, and resumes when it is run again.

Method notes#

  • The reference is the majority vote of the expert labelers. Of 1,633 items with expert votes, 1,451 have a majority, and the rest are left out.
  • Confidence intervals come from 2,000 bootstrap replicates that resample questions. Items that share a question are not independent, so resampling items would give intervals that are too narrow.
  • A segment is a drift zone when its whole kappa interval is below the overall kappa.

Result#

The published table reproduces#

The reproduction step matches all 8 cells of the published table, in the percentage and in the vote count. So the code reads the labels the way the authors did. The protocol runs this check before it measures a new judge.

Table 5 of Zheng et al. (2023), recomputed from the released votesAgreement as the paper defines it. S1 and S2 are the two setups of the paper. In S1, ties count.
Turn Pair of raters Setup Paper This code Votes, paper / this code Match
1 GPT-4 pairwise and human S1 66% 66.0% 1343 / 1343 yes
1 GPT-4 pairwise and human S2 85% 84.6% 859 / 859 yes
1 human and human S1 63% 63.0% 721 / 721 yes
1 human and human S2 81% 81.0% 479 / 479 yes
2 GPT-4 pairwise and human S1 66% 65.7% 1325 / 1325 yes
2 GPT-4 pairwise and human S2 85% 84.6% 864 / 864 yes
2 human and human S1 67% 66.6% 707 / 707 yes
2 human and human S2 82% 81.9% 474 / 474 yes

Agreement with the human majority label#

Cohen's kappa for gpt-6.1-sol is 0.58, with a 95% interval of 0.46 to 0.71. The GPT-4 judge of the 2023 release has 0.55 on the same items. The two intervals overlap, so this sample does not establish a difference between the two judges.

Agreement with the human majority label121 MT-Bench items that have a verdict in both answer orders. Intervals are 95%, from 2,000 bootstrap replicates that resample questions.
Judge Items Raw agreement [95% CI] Cohen's kappa [95% CI] Kappa on a scale from 0 to 1
gpt-6.1-sol 121 0.74 [0.66, 0.82] 0.58 [0.46, 0.71]
GPT-4 pairwise, 2023 release, same items 121 0.71 [0.63, 0.79] 0.55 [0.43, 0.66]
A judge that always gives the most common human label 121 0.46 0.00
estimate95% interval

A judge that always gives the most common human label scores 0.46 raw agreement and 0.00 kappa. Raw agreement alone hides that baseline.

Ties are the weak class#

For gpt-6.1-sol, tie precision is 0.56 and tie recall is 0.45. Win precision is 0.77 and win recall is 0.80. The judge gives a tie on 16 items, and the humans on 20.

gpt-6.1-sol, 121 items
judge: firstjudge: secondjudge: tie
human: first4763
human: second7344
human: tie749
GPT-4 pairwise, 2023 release, the same 121 items
judge: firstjudge: secondjudge: tie
human: first4259
human: second6327
human: tie4412

Rows are the human majority label and columns are the judge verdict. Each item names its two models in a fixed order, and "first" and "second" say which of them wins, whatever order the answers were shown in. Filled cells are agreements.

Precision and recall per classThe two win labels are pooled into one class. A win for the wrong model is not a hit.
Judge Class Precision Judge verdicts Recall Human labels
gpt-6.1-sol win 0.77 105 0.80 101
gpt-6.1-sol tie 0.56 16 0.45 20
GPT-4 pairwise, 2023 win 0.80 93 0.73 101
GPT-4 pairwise, 2023 tie 0.43 28 0.60 20

No category is a drift zone, and the sample is too small to find one#

Coding has the lowest kappa at 0.37 [0.11, 0.63]. With 20 items per category, the intervals are too wide to separate categories. The dashed line marks the overall kappa, and no interval lies wholly below it.

gpt-6.1-sol by question categoryA category is a drift zone when its whole kappa interval is below the overall kappa of 0.58. No category meets that rule.
Category Judged in both orders, of 20 sampled Raw agreement Kappa [95% CI] Kappa on a scale from 0 to 1
coding 20 of 20 0.65 0.37 [0.11, 0.63]
humanities 18 of 20 0.67 0.44 [0.07, 0.75]
math 20 of 20 0.65 0.44 [0.10, 0.82]
stem 20 of 20 0.80 0.62 [0.31, 0.90]
extraction 20 of 20 0.80 0.69 [0.35, 0.92]
reasoning 20 of 20 0.85 0.76 [0.45, 1.00]
roleplay 3 of 20 1.00 1.00 [1.00, 1.00]
writing 0 of 20 no estimate no estimate
estimate95% intervaloverall kappa, 0.58
gpt-6.1-sol by turn of the conversation
Turn Items Raw agreement Kappa [95% CI] Kappa on a scale from 0 to 1
1 60 0.72 0.54 [0.34, 0.73]
2 61 0.77 0.62 [0.44, 0.79]

Answer order#

The verdict is the same in both answer orders for 113 of 121 items (93%). Of the 8 that differ, 5 move toward the answer shown first and 3 toward the answer shown second.

Reproduce it#

The commands are the ones in the README. fetch, reproduce, sample and report make no model call. report on the committed run reproduces the committed report.

python -m pip install -e ".[test]"
python -m pytest -q

judge-calibration fetch
judge-calibration reproduce
judge-calibration --run runs/my-judge sample --per-stratum 10
judge-calibration --run runs/my-judge judge --model MODEL_NAME
judge-calibration --run runs/my-judge report

The full report of the committed run is the file runs/gpt-6.1-sol/report.md in the repository.

Data and credits#

  • Human votes and GPT-4 verdicts: lmsys/mt_bench_human_judgments on Hugging Face, CC BY 4.0.
  • Judge prompts, questions and reference answers: FastChat at commit 587d5cf, Apache-2.0.
  • Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023, arXiv:2306.05685.

Limits#

  • The run is incomplete. It stopped at 243 of 320 calls to limit its cost. Writing has 0 of 20 items and roleplay has 3 of 20, so the results say nothing about writing.
  • The sample does not separate the two judges. The kappa intervals of gpt-6.1-sol and of the GPT-4 judge of 2023 overlap.
  • The sample does not separate categories. With 20 items per category the intervals are too wide, so the finding that no category is a drift zone says little.
  • The reference is noisy. The human labelers agree with each other on 63% to 67% of vote pairs when ties count (Zheng et al., Table 5). That limits the agreement a judge can reach. In this sample, 68 of the 121 judged items have one human vote.
  • The setup differs from FastChat. The system prompt and the prompt template are FastChat's, unchanged, but they are sent together as one user message, because the judge runs inside a Hermes Agent session that has its own system prompt. The run does not set the temperature, and FastChat sets it to 0. The reasoning setting is low.
  • One run. The repository holds one calibration run: gpt-6.1-sol on MT-Bench.