GoKawiilImagine evaluating a retrieval-augmented-generation system. A user asks a question, the system retrieves a text passage, and an LLM judge decides whether it’s relevant. To reduce noise, you ask several judge models to evaluate the same passage. Eight say “relevant”; two say “not relevant”.
Eight out of 10 feels convincing. But the important question is not only how many judges agreed but how independently they arrived at that agreement.
If the eight agreeing judges are genuinely different sources of evidence, then agreement is a strong signal. But if they share a prompt template, a training lineage, a model family, or a common blind spot, they may be repeating the same mistake. The vote count makes the evidence look stronger than it really is.
Correlation between different judges' outputs limits the utility of multijudge panels.
Our paper “Dependence-aware label aggregation for LLM-as-a-judge via Ising models,” coauthored with Shiva Kasiviswanathan and presented at this year’s International Conference on Machine Learning (ICML), addresses this problem. We present a method for assessing the correlations between judges’ outputs and adjusting the aggregate score accordingly, to ensure a diversity of opinion.
In tests on three different tasks, our method outperformed the best-performing baseline — a panel of judges weighted according to historical accuracy — by 9% to 14% on standard metrics.
Ten votes may contain fewer than ten independent pieces of evidence when some judges' outputs are correlated.
Hidden assumptions
The attraction of majority vote is its simplicity. Every judge gets one vote, and the answer with more votes wins. Weighted majority vote is a natural improvement: judges that appear more accurate get more influence.
Both approaches are useful baselines. But they are built around the same simplified view of the judge panel: judges that get the wrong answer are treated as though they make their errors independently.
... continue reading