Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detection
Dude is the first dual-detection multi-agent system designed to identify discrepancies between academic papers and their associated code implementations The system addresses granularity asymmetry between natural language papers and code, which causes over-interpretation and over-reporting of false positives in multi-agent designs A granularity-aligned negotiation mechanism and a two-stage salience-filtering mechanism are introduced to reduce false discrepancy reports Experimental results show up
Analysis
TL;DR
- Dude is the first dual-detection multi-agent system designed to identify discrepancies between academic papers and their associated code implementations
- The system addresses granularity asymmetry between natural language papers and code, which causes over-interpretation and over-reporting of false positives in multi-agent designs
- A granularity-aligned negotiation mechanism and a two-stage salience-filtering mechanism are introduced to reduce false discrepancy reports
- Experimental results show up to 22.8% improvement in recall and precision, and up to 18.7% increase in F1 score compared to baseline methods
- The work is motivated by the growing volume of research submissions that exceed manual review capacity, making LLM-empowered automated detection increasingly necessary
Why It Matters
As AI research submissions continue to scale, manually verifying that published code matches paper claims has become infeasible, creating a critical need for automated discrepancy detection. This work directly addresses a gap in AI research integrity by introducing a multi-agent architecture that improves both recall and precision, which is essential for trustworthy automated peer review and reproducibility verification.
Technical Details
- Dual-Detection Multi-Agent Architecture: Dude employs two complementary detection agents that cross-validate findings, addressing the one-sided detection limitations of single-agent LLM paradigms
- Granularity-Alignment Negotiation: The system introduces a negotiation mechanism that aligns the different granularity levels between paper language (high-level descriptions) and code (low-level implementations), reducing over-interpretation errors
- Two-Stage Salience-Filtering Mechanism: A filtering pipeline that first identifies potentially discrepant claims and then re-evaluates their salience to suppress false positives before final reporting
- Benchmark Performance: Evaluated on real-world paper-code discrepancy datasets, achieving up to 22.8% improvement in both recall and precision, and up to 18.7% F1 score gain over baselines
- Context Capacity Consideration: The multi-agent design accounts for limited context windows in LLMs, distributing the detection task across agents to overcome single-agent context constraints
Industry Insight
- The increasing volume of AI research submissions makes automated verification tools like Dude essential for maintaining research integrity and reproducibility standards in the community
- Multi-agent LLM systems that address granularity mismatches between modalities (text vs. code) represent a generalizable pattern applicable to other cross-modal verification tasks beyond paper-code comparison
- The demonstrated improvement in both precision and recall suggests that dual-detection architectures could become a standard pattern for automated peer review and code-paper validation pipelines in academic publishing
Disclaimer: The above content is generated by AI and is for reference only.