Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
Introduces LLM-NRM, an option-aware psychometric framework that models the full distribution over answer choices rather than binary correct/incorrect scoring Demonstrates that incorrect responses carry systematic information about LLM behavior, with distractor identity contributing +101% additional Fisher Information beyond correctness alone Achieves Spearman correlation of 0.920 with external human-preference Elo leaderboard and 0.943 when using incorrect responses alone for ability estimation
Analysis
TL;DR
- Introduces LLM-NRM, an option-aware psychometric framework that models the full distribution over answer choices rather than binary correct/incorrect scoring
- Demonstrates that incorrect responses carry systematic information about LLM behavior, with distractor identity contributing +101% additional Fisher Information beyond correctness alone
- Achieves Spearman correlation of 0.920 with external human-preference Elo leaderboard and 0.943 when using incorrect responses alone for ability estimation
- Enables efficient benchmarking: 41 selected items preserve full-bank ranking (Kendall's 0.85), achieving a 770x reduction in evaluation cost
- Validated across 189 LLMs, 31,554 items from 14 benchmarks, outperforming both binary Item Response models and conventional nominal-response baselines
Why It Matters
This work fundamentally challenges the binary scoring paradigm that dominates LLM evaluation, showing that the pattern of wrong answers is as informative as correctness itself. For AI practitioners, this means benchmark results can be dramatically compressed without losing ranking fidelity, reducing evaluation costs by orders of magnitude. For researchers, it opens new avenues for diagnosing model behavior through psychometric analysis of option-level preferences.
Technical Details
- LLM-NRM Framework: Extends the classical Nominal Response Model from psychometrics to LLM evaluation, jointly estimating model ability and option-level item characteristics while disentangling response calibration sharpness, positional preference, and difficulty-dependent fallback behavior
- Evaluation Scale: Tested across 189 LLMs and 31,554 items spanning 14 diverse multiple-choice benchmarks
- Information-Theoretic Analysis: Quantified distractor identity contribution as +101% additional Fisher Information per item beyond binary correctness, demonstrating that wrong-answer patterns are highly informative
- Benchmark Compression: Learned item parameters enable subset selection, where only 41 items (from the full bank) preserve ranking with Kendall's correlation of 0.85, yielding 770x evaluation reduction
- Superior Predictive Performance: LLM-NRM outperforms both binary Item Response Theory models and conventional nominal-response baselines on held-out LLM-item interaction prediction
Industry Insight
- Benchmark design should evolve from binary scoring to option-aware psychometric frameworks that extract maximum signal from every response, potentially becoming the new standard for reliable LLM evaluation
- The 770x compression ratio suggests that expensive full-bank evaluations may be replaceable by carefully selected item subsets, enabling faster iteration cycles and more frequent model assessment
- The strong correlation (0.920) with human-preference Elo rankings validates that psychometric ability estimates align closely with perceived model quality, giving practitioners a theoretically grounded alternative to raw accuracy metrics
Disclaimer: The above content is generated by AI and is for reference only.