Evals/Area 1 of 5/Judge reliability/P2
Verdict aggregation
When several judges vote on the same item, how should the votes become one verdict? Majority vote against the Dawid-Skene model and an item response model.
Headline result
1,184of1,517
gold items of one dataset have no vote in the released files, and the authors' scoring code gives each of them a random label
The result tables- Data
- Four public crowdsourcing datasets, Zheng et al. (VLDB 2017)
- Sample
- 1,000 to 8,315 gold items per dataset
- Model calls
- None
- Checks in CI
- Tests, the reproduction of the published table, and both prototypes
- Repository
- verdict-aggregation
The question#
When several judges vote on the same item, how do you turn the votes into one verdict?
I evaluate AI agents with panels of LLM judges, and the usual answer is a majority vote. The statistics literature has a second answer from 1979: the Dawid-Skene model, which learns how reliable each voter is and weights the votes. I wanted to know when the model is worth the extra work.
What I built#
- Majority voting and Dawid-Skene in about 50 lines of NumPy.
- A reproduction of the published results for both methods on four public crowdsourcing datasets, from Zheng et al., "Truth Inference in Crowdsourcing: Is the Problem Solved?" (VLDB 2017).
- A simulation of judge panels for the case that matters for LLM judges: judges that make the same mistakes.
- Two prototypes: an item response model for votes, and a split of run-to-run noise into an agent part and a judge part. Both are first versions. The code runs and has tests.
No step calls a model.
Method notes#
- Dawid-Skene gives each voter a confusion matrix: the chance of each vote for each true label. Expectation maximization estimates the matrices and the true labels together. I use the settings of the authors: 20 iterations, and a start value of 0.7 on the diagonal.
- Scoring follows the code of the authors. A tied item gets a random label among the tied ones. Accuracy is the exact expected value of that rule, and ± is its standard deviation over 1,000 seeded draws.
- GLAD (Whitehill et al., 2009) is the item response model for votes. Each voter has one ability, each item has one ease, and a vote is right with a chance that grows with both.
verdict_aggregation/irt.pyimplements it in about 60 lines.
Result#
The published table reproduces#
All 8 accuracy values and all 4 F1 values of Table 6 match. Four of the accuracy values match exactly. In the other four, the scoring rule of the authors makes random choices, and the published value is within two standard deviations of my result.
| Dataset | Method | Accuracy, paper | Accuracy, this code | F1, paper | F1, this code | Items scored at random | Match |
|---|---|---|---|---|---|---|---|
| D_Product | Majority vote | 89.66 | 89.66 | 59.05 | 59.05 | 0 of 8315 | yes |
| D_Product | Dawid-Skene | 93.66 | 93.66 | 71.59 | 71.59 | 0 of 8315 | yes |
| D_PosSent | Majority vote | 93.31 | 93.35 ± 0.33 | 92.85 | 92.85 ± 0.36 | 43 of 1000 | yes |
| D_PosSent | Dawid-Skene | 96.00 | 96.00 | 95.66 | 95.66 | 0 of 1000 | yes |
| S_Rel | Majority vote | 54.19 | 54.11 ± 0.30 | n/a | n/a | 916 of 4460 | yes |
| S_Rel | Dawid-Skene | 61.30 | 61.30 | n/a | n/a | 0 of 4460 | yes |
| S_Adult | Majority vote | 36.04 | 36.19 ± 0.94 | n/a | n/a | 1195 of 1517 | yes |
| S_Adult | Dawid-Skene | 36.05 | 35.86 ± 0.94 | n/a | n/a | 1184 of 1517 | yes |
One of the four results measures random guessing#
The paper reports about 36% accuracy for every method on S_Adult, and it concludes that the methods perform alike there.
In the released files, 1,184 of the 1,517 gold items of S_Adult have no vote at all. The scoring code of the authors gives each of those items a random label out of four. So 78% of the scored items get a random label.
| Method | All 1,517 gold items | The 333 gold items that have votes | Accuracy on a scale from 0 to 100 |
|---|---|---|---|
| Majority vote | 36.19 | 75.98 | |
| Dawid-Skene | 35.86 | 74.47 |
The arithmetic agrees: (333 × 0.7598 + 1,184 × 0.25) / 1,517 = 0.362.
I checked the raw source files in the archive (gold.txt and labels.txt), and the overlap is the same 333 items. This is a statement about the released archive. I have not asked the authors if their own copy of the data differs.
Dawid-Skene needs more than one vote per item, then it wins#
With one vote per item, the model has nothing to compare, and it is slightly worse than the majority. From three votes per item, it is ahead on both datasets I tested.
| Dataset | Votes per item | Majority vote | Dawid-Skene | Accuracy on a scale from 0 to 100 |
|---|---|---|---|---|
| D_PosSent | 1 | 68.91 ± 1.73 | 67.84 ± 1.66 | |
| D_PosSent | 3 | 76.61 ± 1.07 | 84.72 ± 1.21 | |
| D_PosSent | 5 | 81.73 ± 0.91 | 91.09 ± 0.73 | |
| D_PosSent | 10 | 87.63 ± 0.50 | 94.51 ± 0.50 | |
| D_PosSent | 20 | 93.35 | 96.00 | |
| S_Rel | 1 | 45.41 ± 0.59 | 43.89 ± 0.53 | |
| S_Rel | 2 | 45.13 ± 0.44 | 51.13 ± 0.73 | |
| S_Rel | 3 | 49.60 ± 0.41 | 56.06 ± 0.49 | |
| S_Rel | 4 | 51.86 ± 0.33 | 58.77 ± 0.60 | |
| S_Rel | 5 | 53.38 ± 0.19 | 60.46 ± 0.33 |
Neither method fixes judges that share their mistakes#
Crowd workers are different people. LLM judges are often the same model run several times, so their errors are not independent. The simulation gives each judge 70% accuracy and varies how much of the error is shared.
| Scenario | Judges | Majority vote | Dawid-Skene | Accuracy on a scale from 0 to 100 |
|---|---|---|---|---|
| Equal judges, independent errors | 3 | 81.29 ± 0.90 | 81.62 ± 0.99 | |
| Equal judges, independent errors | 5 | 88.84 ± 0.61 | 89.71 ± 0.79 | |
| Equal judges, independent errors | 10 | 96.92 ± 0.51 | 97.09 ± 0.54 | |
| One strong judge (90%) among weak ones (55%) | 3 | 78.94 ± 1.22 | 88.97 ± 1.82 | |
| One strong judge (90%) among weak ones (55%) | 5 | 80.18 ± 0.90 | 89.83 ± 0.83 | |
| One strong judge (90%) among weak ones (55%) | 10 | 87.52 ± 0.91 | 92.88 ± 0.84 | |
| Equal judges, half of the errors shared | 3 | 76.32 ± 1.02 | 76.31 ± 1.34 | |
| Equal judges, half of the errors shared | 5 | 78.77 ± 0.88 | 78.74 ± 1.06 | |
| Equal judges, half of the errors shared | 10 | 78.91 ± 0.96 | 78.21 ± 1.02 | |
| Equal judges, all errors shared | 3 | 70.28 ± 1.31 | 70.28 ± 1.31 | |
| Equal judges, all errors shared | 5 | 70.10 ± 1.45 | 70.10 ± 1.45 | |
| Equal judges, all errors shared | 10 | 70.37 ± 1.71 | 70.37 ± 1.71 |
- With equal, independent judges, the two methods are equal. The model has no reliability difference to learn.
- With judges of unequal quality, Dawid-Skene finds the strong judge. With 3 judges it gains 10 points.
- With shared errors, more judges stop helping. At half-shared errors, 10 judges score 79%, and 3 judges score 76%. With independent errors, 10 judges score 97%.
The last point is the one I take to my own work. Ten runs of one judge model are not ten opinions.
Prototype: an item response model for votes#
Dawid-Skene gives each voter a confusion matrix. Item response theory makes a different assumption: one ability per voter and one ease per item.
| Dataset | Votes per gold item | Majority vote | Dawid-Skene | GLAD, this code | GLAD, paper |
|---|---|---|---|---|---|
| D_Product | 3.0 | 89.66 | 93.66 | 92.81 | 92.20 |
| D_PosSent | 20.0 | 93.35 | 96.00 | 94.80 | 95.20 |
| S_Rel | 5.9 | 54.11 | 61.30 | 51.87 | 53.59 |
| S_Adult | 10.0 | 36.19 | 35.86 | 36.19 | 36.47 |
The GLAD column is a new implementation with its own optimizer. It lands within 1.8 points of the paper on every dataset, and it is not a reproduction.
- One number per voter is not enough here. Dawid-Skene is ahead of GLAD on three datasets, by 0.9 to 9.4 points. S_Adult is the exception, and there all three methods sit near 36%.
- The item parameter finds the hard items. On three datasets, the quarter of items that the model calls hardest has the lowest majority vote accuracy. On S_Rel the four quarters are within 4.4 points of each other.
- Only the dataset with 20 votes per item gives a clean order. D_PosSent goes 81.4, 94.8, 98.2, 99.0 from hardest to easiest. D_Product has 3 votes per item, and its "easiest" quarter scores 88.5%, below both middle quarters. A panel of three LLM judges is in the D_Product position.
| Dataset | Votes per gold item | Hardest quarter | Second quarter | Third quarter | Easiest quarter |
|---|---|---|---|---|---|
| D_Product | 3.0 | 78.50 | 97.26 | 94.42 | 88.45 |
| D_PosSent | 20.0 | 81.40 | 94.80 | 98.20 | 99.00 |
| S_Rel | 5.9 | 53.11 | 52.82 | 53.36 | 57.17 |
| S_Adult | 10.0 | 42.26 | 76.51 | 95.18 | 90.36 |
Prototype: run-to-run noise, agent against judge#
An eval repeated on the same items gives a different score each time. Part of that comes from the agent, which answers the same item differently. Part comes from the judge, which scores the same answer differently. verdict_aggregation/variance.py separates the two with a nested analysis of variance, and then prices each kind of repeated run.
Every number in this part comes from a simulation with a fixed seed. No agent and no judge ran.
| Scenario | Items differ | Agent differs between runs | Judge differs between runs | Split of the variance of one verdict | Krippendorff's alpha between 3 judge runs |
|---|---|---|---|---|---|
| Agent noise is largerThe judge reports the wrong outcome on 5% of its calls | 0.1344 | 0.0679 | 0.0478 | 0.74 | |
| Judge noise is largerThe judge reports the wrong outcome on 20% of its calls | 0.0868 | 0.0032 | 0.1600 | 0.32 |
| Items | Agent runs per item | Judge runs per answer | Judge calls | Agent noise is larger | Judge noise is larger | Interval drawn, 10 points either side |
|---|---|---|---|---|---|---|
| 100 | 1 | 1 | 100 | 9.8 | 9.8 | |
| 100 | 1 | 3 | 300 | 9.2 | 7.4 | |
| 100 | 3 | 1 | 300 | 8.2 | 7.4 | |
| 100 | 3 | 3 | 900 | 7.9 | 6.4 | |
| 300 | 1 | 1 | 300 | 5.7 | 5.7 |
- Which run to repeat depends on the split. When the agent is the noisy part, three judge runs per answer move the interval from 9.8 to 9.2 points. Three agent runs move it to 8.2.
- A new agent run also draws a new verdict. That is why repeating the agent is never worse than repeating the judge at the same number of calls.
- New items beat both. 300 items judged once give 5.7 points in both scenarios, for the same 300 calls. Repeats are for the case where the item set is fixed.
The module also reports Krippendorff's alpha between judge runs and a bootstrap interval that resamples whole items.
Reproduce it#
The commands are the ones in the README. report takes about 30 seconds on a laptop. Every random step has a fixed seed, so a second run writes an identical file.
python -m pip install -e ".[test]"
python -m pytest -q
verdict-aggregation fetch # downloads the datasets, checks the SHA-256
verdict-aggregation reproduce # prints the reproduction table, exits 1 on a mismatch
verdict-aggregation report # runs every experiment, writes results/report.md
verdict-aggregation psychometrics # fits the item response model, writes results/psychometrics.md
verdict-aggregation noise # the simulation of agent and judge noise, writes results/noise.mdData and credits#
- Datasets and reference code: Zheng, Li, Li, Shan and Cheng, Truth Inference in Crowdsourcing: Is the Problem Solved?, VLDB 2017.
fetchdownloads their archive, and the repository holds none of the data. - Dawid and Skene, "Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm", Applied Statistics, 1979.
- Whitehill, Ruvolo, Wu, Bergsma and Movellan, "Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise", NIPS 2009.
- Krippendorff, "Computing Krippendorff's Alpha-Reliability", 2011. The test uses the worked example of that paper.
Limits#
- The votes are from crowd workers. No LLM judge voted in any of the four datasets. The part about LLM judges is a simulation, a model with assumptions, and it is not a measurement of real LLM judges.
- The S_Adult finding describes the released archive. I have not asked the authors if their own copy of the data differs.
- GLAD is a new implementation. It has its own optimizer and lands within 1.8 points of the paper. It is not a reproduction.
- The cause of the clean order is open. Whether the order of the quarters depends on the number of votes needs the same dataset at several vote counts.
- The noise split is a simulation. The same split on real agent and judge runs is open.
- The noise table counts judge calls only. An extra agent run also has a cost, and the table leaves it out.
- The scope is narrower than the paper. The paper compares 17 methods on 5 datasets. I reproduce 2 methods on the 4 datasets with categorical labels.