Evals
I build evaluation infrastructure for AI agents, and I measure whether their judges agree with people.
gpt-6.1-sol0.58 [0.46, 0.71]
GPT-4, 20230.55 [0.43, 0.66]
Cohen's kappa of an LLM judge against the human majority label, with its 95% interval. 121 MT-Bench items, the same items for both judges. The intervals overlap, so this sample does not establish a difference.
- P1Does an LLM judge agree with human labels?
- P2How should several verdicts become one?
- P3Which code change caused the behavior change?
- P4Is the task or the grader at fault?
- P5What happens on every commit?
- P6When does a few-shot prompt earn its cost?