Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System
ML peer review is in crisis due to soaring submission volumes, stricter reciprocal review policies, and the absence of publication fees, yet there is almost no constructive public discourse on improving the system The paper identifies two core problems: how to reasonably limit submission volume and how to incentivize good reviewing while discouraging bad reviewing Existing conference mechanisms are assessed for their strengths and shortcomings, with four popular approaches critiqued The authors
Analysis
TL;DR
- ML peer review is in crisis due to soaring submission volumes, stricter reciprocal review policies, and the absence of publication fees, yet there is almost no constructive public discourse on improving the system
- The paper identifies two core problems: how to reasonably limit submission volume and how to incentivize good reviewing while discouraging bad reviewing
- Existing conference mechanisms are assessed for their strengths and shortcomings, with four popular approaches critiqued
- The authors propose "OpenReview Points," a currency-like credit system where practitioners earn points through quality reviewing and spend them on conference perks like complimentary registration or additional review resources
- Meaningful reform requires enforceable, fine-grained procedural safeguards paired with incentive structures rather than polite best-practice suggestions in reviewer guidelines
Why It Matters
This paper directly addresses a systemic pain point affecting every ML researcher and practitioner: the deteriorating quality of peer review. As submission volumes continue to explode and conferences struggle to maintain review standards, the proposed credit system offers a concrete, incentive-based framework that could reshape how the community approaches scholarly evaluation. For AI practitioners, understanding these reform proposals is essential as they may soon become adopted by major venues.
Technical Details
- The paper is a position paper submitted to ICML 2026 (Position Paper Track), authored by Shaochen Zhong, available on arXiv under cs.AI and cs.DL
- It evaluates four existing conference mechanisms aimed at addressing review quality and submission volume, analyzing both their strengths and shortcomings
- The proposed "OpenReview Points" system functions as an internal currency: reviewers earn points by contributing high-quality reviews and can redeem them across one or multiple major conferences
- Perks redeemable with points include complimentary conference registration and the right to request additional review resources, creating a tangible incentive structure
- The framework pairs these economic incentives with "enforceable yet fine-grained procedural safeguards" to ensure accountability and quality control in the review process
Industry Insight
- The ML community should anticipate a shift from voluntary, goodwill-based reviewing toward structured incentive systems; practitioners who actively contribute quality reviews will gain tangible career benefits, making reviewing a strategic investment rather than a burden
- Conference organizers and platform providers like OpenReview should begin piloting credit-based systems, as the current trajectory of increasing submissions and declining review quality makes incremental policy changes insufficient
- Researchers should monitor this reform movement closely, as the adoption of point-based review systems could reshape academic incentives, conference economics, and the distribution of review labor across the community within the next few years
Disclaimer: The above content is generated by AI and is for reference only.