When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
The study challenges the fundamental assumption in LLM-as-judge systems that self-consistency or cross-model agreement reliably indicates correctness. Analysis of 265,000 samples reveals that agreement is a weak predictor of accuracy (rho 0.20-0.59) and often stems from shared biases or memorized heuristics rather than truth. Frontier models exhibit dangerous over-confidence, maintaining high agreement rates (>0.8) even when nearly half of those agreements are incorrect. Self-consistency serves
Analysis
TL;DR
- The study challenges the fundamental assumption in LLM-as-judge systems that self-consistency or cross-model agreement reliably indicates correctness.
- Analysis of 265,000 samples reveals that agreement is a weak predictor of accuracy (rho 0.20-0.59) and often stems from shared biases or memorized heuristics rather than truth.
- Frontier models exhibit dangerous over-confidence, maintaining high agreement rates (>0.8) even when nearly half of those agreements are incorrect.
- Self-consistency serves only as a conditional proxy for correctness, being most useful for mid-tier models and compute allocation, but unreliable for high-end systems.
Why It Matters
This research critically undermines the reliability of current evaluation pipelines that depend on consensus mechanisms, such as ensemble judging or self-consistency decoding, to verify AI outputs. For practitioners, it signals that high agreement scores should not be interpreted as high confidence in factual correctness, particularly when deploying state-of-the-art models. This necessitates a shift toward more robust verification methods that account for systemic biases and hallucination patterns common across different model architectures.
Technical Details
- Dataset and Scale: The study utilized a large-scale cross-runner experiment involving 53 independent runners generating K=50 samples each on GPQA Diamond and AIME benchmarks, totaling 265,000 samples.
- Methodology: Researchers employed majority-correctness as the ground truth label and used hierarchical runner-clustered bootstrapping to assess the correlation between agreement and accuracy.
- Key Findings: Agreement showed a positive but weak correlation with correctness (Spearman’s rho 0.20-0.59). Frontiers models agreed on >=0.8 of GPQA entries 77% of the time, yet were wrong in 48% of those instances.
- Cross-Family Validation: An exploratory check across three tiers of Claude models confirmed that confident errors recur across providers, indicating that the issue is not isolated to a single architecture but is a broader phenomenon among top-tier models.
Industry Insight
- Rethink Evaluation Metrics: Organizations relying on LLM-as-judge frameworks for automated grading or quality assurance must implement additional validation layers, such as human-in-the-loop checks or external knowledge grounding, rather than trusting consensus alone.
- Compute Allocation Strategy: While self-consistency is poor for verifying frontier model outputs, it remains a viable heuristic for optimizing inference costs on mid-tier models where agreement correlates better with correctness.
- Bias Awareness: Developers should audit their prompt engineering and model selection processes for shared biases, as different models may converge on incorrect answers due to similar training data artifacts or positional priors.
Disclaimer: The above content is generated by AI and is for reference only.