GoKawiilAmazon researchers unveil Ising-model method to fix correlated LLM judge panels
Amazon Science researchers, together with Shiva Kasiviswanathan, presented a new paper at ICML that tackles a flaw in using multiple LLM judges to evaluate outputs like retrieval passages: when judges share prompts, training data, or model lineage, their agreement can look stronger than it actually is. Their method models correlations between judges using Ising models and reweights votes to account for shared blind spots rather than treating every vote as independent evidence.