GoKawiilAmazon researchers unveil Ising-model method to fix correlated LLM judge panels
Amazon Science researchers, together with Shiva Kasiviswanathan, presented a new paper at ICML that tackles a flaw in using multiple LLM judges to evaluate outputs like retrieval passages: when judges share prompts, training data, or model lineage, their agreement can look stronger than it actually is. Their method models correlations between judges using Ising models and reweights votes to account for shared blind spots rather than treating every vote as independent evidence.
GoKawiil's interpretation of the reporting above, not reported fact.
Many AI evaluation pipelines rely on panels of LLM judges to reduce noise, but if those judges are not truly independent, majority or accuracy-weighted voting can systematically overstate confidence in flawed judgments. This work offers a statistically grounded way to detect and correct for that hidden correlation, improving the reliability of automated evaluation systems used across AI development.
- Majority vote among LLM judges can overstate confidence when judges share biases or training lineage.
- The new Ising-model-based method adjusts aggregate scores by modeling dependence between judges.
- It outperformed the best baseline, an accuracy-weighted judge panel, by 9% to 14% on three tasks.
Source: amazon.science — About The Author, 2026-09-14
Published there as: “When LLM judges agree, should we believe them?”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.