Evals/Area 1 of 5/Judge reliability/P1
Judge calibration
How far can I trust an LLM judge? A calibration protocol against human labels, written so that another person can run it again.
Headline result
0.58
Cohen's kappa of the judge against the human majority label, 95% interval 0.46 to 0.71
The result tables- Data
- MT-Bench human judgments, Zheng et al. (2023)
- Sample
- 121 items with a verdict in both answer orders, of 160 sampled
- Model calls
- 243 of 320, then the run stopped
- Checks in CI
- Tests, and the reproduction of the published table from the pinned public data
- Repository
- judge-calibration
MIT license
The question#
How far can I trust this judge? The protocol answers that one question. It reports agreement with human labels above the chance level, precision and recall per class, the segments where the judge is weaker, and how often the verdict changes with the order of the answers.
The example calibrates gpt-6.1-sol as a pairwise judge on MT-Bench. The reference is the expert human votes that Zheng et al. released in 2023.
What I built#
A command line tool with five steps. Four of them make no model call.
fetchdownloads the human votes, the questions and the unchanged FastChat judge prompts. Each file is pinned by commit and SHA-256.reproducerecomputes Table 5 of Zheng et al. (2023) from the released votes. A calibration on data that the code reads wrongly is worthless, so this step runs in CI.sampledraws a seeded sample of items that have a human majority label, with the same number per category and turn.judgeshows each item to the judge twice, once per answer order. A win counts only when the judge names the same model in both orders. This is FastChat's rule.reportwrites the statistics toreport.mdandreport.json.
How the judge is called#
judge calls the model through Hermes Agent with the openai-codex provider. Each call is a fresh session with user configuration, memory and rules off. ask_hermes in judge_calibration/judge.py is the one function to replace for another backend.
The command appends each result at once, stops after five failed calls in a row, and resumes when it is run again.
Method notes#
- The reference is the majority vote of the expert labelers. Of 1,633 items with expert votes, 1,451 have a majority, and the rest are left out.
- Confidence intervals come from 2,000 bootstrap replicates that resample questions. Items that share a question are not independent, so resampling items would give intervals that are too narrow.
- A segment is a drift zone when its whole kappa interval is below the overall kappa.
Result#
The published table reproduces#
The reproduction step matches all 8 cells of the published table, in the percentage and in the vote count. So the code reads the labels the way the authors did. The protocol runs this check before it measures a new judge.
| Turn | Pair of raters | Setup | Paper | This code | Votes, paper / this code | Match |
|---|---|---|---|---|---|---|
| 1 | GPT-4 pairwise and human | S1 | 66% | 66.0% | 1343 / 1343 | yes |
| 1 | GPT-4 pairwise and human | S2 | 85% | 84.6% | 859 / 859 | yes |
| 1 | human and human | S1 | 63% | 63.0% | 721 / 721 | yes |
| 1 | human and human | S2 | 81% | 81.0% | 479 / 479 | yes |
| 2 | GPT-4 pairwise and human | S1 | 66% | 65.7% | 1325 / 1325 | yes |
| 2 | GPT-4 pairwise and human | S2 | 85% | 84.6% | 864 / 864 | yes |
| 2 | human and human | S1 | 67% | 66.6% | 707 / 707 | yes |
| 2 | human and human | S2 | 82% | 81.9% | 474 / 474 | yes |
Agreement with the human majority label#
Cohen's kappa for gpt-6.1-sol is 0.58, with a 95% interval of 0.46 to 0.71. The GPT-4 judge of the 2023 release has 0.55 on the same items. The two intervals overlap, so this sample does not establish a difference between the two judges.
| Judge | Items | Raw agreement [95% CI] | Cohen's kappa [95% CI] | Kappa on a scale from 0 to 1 |
|---|---|---|---|---|
gpt-6.1-sol |
121 | 0.74 [0.66, 0.82] | 0.58 [0.46, 0.71] | |
| GPT-4 pairwise, 2023 release, same items | 121 | 0.71 [0.63, 0.79] | 0.55 [0.43, 0.66] | |
| A judge that always gives the most common human label | 121 | 0.46 | 0.00 |
A judge that always gives the most common human label scores 0.46 raw agreement and 0.00 kappa. Raw agreement alone hides that baseline.
Ties are the weak class#
For gpt-6.1-sol, tie precision is 0.56 and tie recall is 0.45. Win precision is 0.77 and win recall is 0.80. The judge gives a tie on 16 items, and the humans on 20.
| judge: first | judge: second | judge: tie | |
|---|---|---|---|
| human: first | 47 | 6 | 3 |
| human: second | 7 | 34 | 4 |
| human: tie | 7 | 4 | 9 |
| judge: first | judge: second | judge: tie | |
|---|---|---|---|
| human: first | 42 | 5 | 9 |
| human: second | 6 | 32 | 7 |
| human: tie | 4 | 4 | 12 |
Rows are the human majority label and columns are the judge verdict. Each item names its two models in a fixed order, and "first" and "second" say which of them wins, whatever order the answers were shown in. Filled cells are agreements.
| Judge | Class | Precision | Judge verdicts | Recall | Human labels |
|---|---|---|---|---|---|
gpt-6.1-sol |
win | 0.77 | 105 | 0.80 | 101 |
gpt-6.1-sol |
tie | 0.56 | 16 | 0.45 | 20 |
| GPT-4 pairwise, 2023 | win | 0.80 | 93 | 0.73 | 101 |
| GPT-4 pairwise, 2023 | tie | 0.43 | 28 | 0.60 | 20 |
No category is a drift zone, and the sample is too small to find one#
Coding has the lowest kappa at 0.37 [0.11, 0.63]. With 20 items per category, the intervals are too wide to separate categories. The dashed line marks the overall kappa, and no interval lies wholly below it.
gpt-6.1-sol by question categoryA category is a drift zone when its whole kappa interval is below the overall kappa of 0.58. No category meets that rule.| Category | Judged in both orders, of 20 sampled | Raw agreement | Kappa [95% CI] | Kappa on a scale from 0 to 1 |
|---|---|---|---|---|
| coding | 20 of 20 | 0.65 | 0.37 [0.11, 0.63] | |
| humanities | 18 of 20 | 0.67 | 0.44 [0.07, 0.75] | |
| math | 20 of 20 | 0.65 | 0.44 [0.10, 0.82] | |
| stem | 20 of 20 | 0.80 | 0.62 [0.31, 0.90] | |
| extraction | 20 of 20 | 0.80 | 0.69 [0.35, 0.92] | |
| reasoning | 20 of 20 | 0.85 | 0.76 [0.45, 1.00] | |
| roleplay | 3 of 20 | 1.00 | 1.00 [1.00, 1.00] | |
| writing | 0 of 20 | no estimate | no estimate |
gpt-6.1-sol by turn of the conversation| Turn | Items | Raw agreement | Kappa [95% CI] | Kappa on a scale from 0 to 1 |
|---|---|---|---|---|
| 1 | 60 | 0.72 | 0.54 [0.34, 0.73] | |
| 2 | 61 | 0.77 | 0.62 [0.44, 0.79] |
Answer order#
The verdict is the same in both answer orders for 113 of 121 items (93%). Of the 8 that differ, 5 move toward the answer shown first and 3 toward the answer shown second.
Reproduce it#
The commands are the ones in the README. fetch, reproduce, sample and report make no model call. report on the committed run reproduces the committed report.
python -m pip install -e ".[test]"
python -m pytest -q
judge-calibration fetch
judge-calibration reproduce
judge-calibration --run runs/my-judge sample --per-stratum 10
judge-calibration --run runs/my-judge judge --model MODEL_NAME
judge-calibration --run runs/my-judge reportThe full report of the committed run is the file runs/gpt-6.1-sol/report.md in the repository.
Data and credits#
- Human votes and GPT-4 verdicts:
lmsys/mt_bench_human_judgmentson Hugging Face, CC BY 4.0. - Judge prompts, questions and reference answers: FastChat at commit
587d5cf, Apache-2.0. - Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023, arXiv:2306.05685.
Limits#
- The run is incomplete. It stopped at 243 of 320 calls to limit its cost. Writing has 0 of 20 items and roleplay has 3 of 20, so the results say nothing about writing.
- The sample does not separate the two judges. The kappa intervals of
gpt-6.1-soland of the GPT-4 judge of 2023 overlap. - The sample does not separate categories. With 20 items per category the intervals are too wide, so the finding that no category is a drift zone says little.
- The reference is noisy. The human labelers agree with each other on 63% to 67% of vote pairs when ties count (Zheng et al., Table 5). That limits the agreement a judge can reach. In this sample, 68 of the 121 judged items have one human vote.
- The setup differs from FastChat. The system prompt and the prompt template are FastChat's, unchanged, but they are sent together as one user message, because the judge runs inside a Hermes Agent session that has its own system prompt. The run does not set the temperature, and FastChat sets it to 0. The reasoning setting is
low. - One run. The repository holds one calibration run:
gpt-6.1-solon MT-Bench.