Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards
Benchmark contamination (test data leaking into training) inflates absolute LLM scores but rarely changes model rankings on leaderboards The authors introduce an anchor-item invariance audit using paraphrased items to isolate memorization from genuine capability Rank correlation between standard and paraphrase-controlled leaderboards is 0.997, indicating minimal reordering impact Contamination among public models is largely uniform; ranking distortion requires rare differential contamination The
Analysis
TL;DR
- Benchmark contamination (test data leaking into training) inflates absolute LLM scores but rarely changes model rankings on leaderboards
- The authors introduce an anchor-item invariance audit using paraphrased items to isolate memorization from genuine capability
- Rank correlation between standard and paraphrase-controlled leaderboards is 0.997, indicating minimal reordering impact
- Contamination among public models is largely uniform; ranking distortion requires rare differential contamination
- The paper releases a calibrated invariance audit as a reference implementation and recommends paraphrase-controlled rankings with confidence intervals
Why It Matters
This research directly challenges the prevailing narrative that benchmark contamination is a primary threat to LLM leaderboard reliability, showing instead that its main effect is score inflation rather than ranking manipulation. For AI practitioners and researchers, it provides a practical, calibrated methodology to audit contamination and a stronger empirical basis for interpreting leaderboard comparisons. The findings are especially relevant as the community debates benchmark integrity and the need for more robust evaluation protocols.
Technical Details
- Method: Contamination is recast as a violation of anchor-item invariance, measured through differential functioning between original and semantically equivalent paraphrased items, holding the measured skill fixed while isolating memorization from capability
- Datasets & Models: Evaluated 47 publicly released models and 74 finetuned models with known contamination doses across four benchmarks: ARC, GSM8K, HellaSwag, and MMLU
- Calibration Results: The measure recovers injected contamination dose-responsively, with a corrected effect of +0.187 accuracy points for test-set leakage, and correctly flags zero contamination for a negative-control model (-0.012)
- Leaderboard Impact: Rank correlation of 0.997 between standard and paraphrase-controlled leaderboards; only 3 of 188 model-by-benchmark cases showed corroborated differential contamination across two references
- Deliverables: A calibrated invariance audit released as a reference implementation, with recommendations for leaderboards to report paraphrase-controlled rankings alongside confidence intervals
Industry Insight
- Leaderboard comparisons should be interpreted with more confidence than commonly assumed; contamination is unlikely to be the driver of rank changes, so observed ranking shifts more likely reflect genuine capability differences
- Benchmark providers and leaderboard operators should adopt paraphrase-controlled evaluations and report confidence intervals to increase transparency and trust in rankings
- The uniform nature of contamination across public models suggests a systemic issue rather than a selective one, pointing toward the need for cleaner benchmark construction rather than model-specific audits
Disclaimer: The above content is generated by AI and is for reference only.