Debate Helps Weak Judges Reward Stronger Models

This research article investigates debate as a scalable oversight protocol, moving beyond theoretical promise to establish clear empirical boundaries for its effectiveness. The core contribution is a set of pre-deployment conditions that predict when debate will actually help a weaker judge oversee more capable debaters on verifiable tasks.

The Prerequisite of Superior Classification

The study identifies a non-obvious performance threshold for the critic model. For debate to outperform consultancy, the critic's ability to correctly classify solutions as correct or incorrect must be statistically superior to the judge's own classification ability. This is a precise, measurable condition:

• Performance is binary: Debate's utility hinges on the critic exceeding a clear capability threshold relative to the judge.
• The capability gap is pairing-specific: The authors find statistically significant gains only in the three of five model pairings where this condition holds, which correspond to the most capable overall pairings.
• Failure mode: In the two pairings where the critic's classification ability is within noise of the judge's, debate produces null effects. The critic's disagreement then becomes noise rather than a useful signal.

The Shift from Testimony to Verifiable Claim

The judge's interpretive framework is a critical, second-order factor. Even with a superior critic, debate fails if the judge does not adopt the right epistemic stance toward the critic's output.

• The required mindset: The judge must parse the critic's speech as a claim to be verified, not as testimony to be summarized. This transforms the process from accepting an authority's word to actively investigating a specific, falsifiable assertion.
• Observable degradation: In non-responder pairings, the mere presence of a critic in the transcript causes judge verification rates to drop by tens of percentage points. This suggests the critic's speech is misinterpreted as conclusive testimony, discouraging independent verification.

A Cheaper Oversight Primitive Emerges

The ablation study reveals a practical and cost-efficient insight. Adding rebuttal rounds to the debate protocol—where the proposer responds to the critic and the critic can counter—yields no measurable improvement in judge performance.

• The core benefit is captured cheaply: A single, independent critique from the critic recovers the bulk of the benefit observed in full multi-round debate. This points to a simpler primitive: (answer, critique, judge).
• Implications for deployment: This finding directly addresses the high inference cost of debate, suggesting that verifiable domains can achieve scalable oversight with significantly less computational overhead.

A Practical Audit for Pre-Deployment

The research culminates in an actionable checklist for determining if debate is a viable oversight tool for a given system:

1. Does the critic beat the judge? Measure classification accuracy on a held-out set for the specific critic-judge model pair.
2. Will the judge verify it? Ensure the judge is prompted or fine-tuned to treat critiques as claims requiring independent verification, not as final answers.

This work shifts the discourse from "does debate work?" to "under what precise conditions does debate work?", providing the field with a clear, empirical roadmap for implementation in verifiable domains like code and logic.