RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving
RePro is the first framework to integrate Lean-oriented neural automated theorem provers (ATPs) into benchmark rewriting for LLM mathematical problem solving evaluation It addresses data contamination by rewriting problems and regenerating answers with correctness guaranteed by Lean-verified proofs Experiments on GSM8K and MATH demonstrate that RePro's retained rewritten instances achieve 100% well-definedness, feasibility, and answer correctness, unlike existing rewriting methods Several models
Analysis
TL;DR
- RePro is the first framework to integrate Lean-oriented neural automated theorem provers (ATPs) into benchmark rewriting for LLM mathematical problem solving evaluation
- It addresses data contamination by rewriting problems and regenerating answers with correctness guaranteed by Lean-verified proofs
- Experiments on GSM8K and MATH demonstrate that RePro's retained rewritten instances achieve 100% well-definedness, feasibility, and answer correctness, unlike existing rewriting methods
- Several models show accuracy drops on proof-verified rewritten benchmarks, suggesting their original performance may partly reflect memorization rather than genuine reasoning ability
Why It Matters
Data contamination has long been a critical threat to the reliability of LLM evaluations, especially in mathematical problem solving where benchmark datasets may leak into training data. RePro introduces a novel verification layer using Lean theorem proving that ensures rewritten benchmarks are both valid and correct, providing a more trustworthy standard for evaluating genuine reasoning capabilities. This is particularly relevant as the field moves toward more rigorous and contamination-resistant evaluation practices.
Technical Details
- Framework: RePro integrates Lean-oriented neural automated theorem provers (ATPs) into the benchmark rewriting pipeline, combining neural rewriting with formal proof verification
- Verification mechanism: Rewritten problems and regenerated answers are validated through Lean-verified proofs, ensuring mathematical correctness at a formal level rather than relying on heuristic checks
- Benchmarks evaluated: GSM8K and MATH datasets were used to test the framework's effectiveness
- Quality guarantees: RePro achieves 100% well-definedness, feasibility, and answer correctness on retained rewritten instances, while existing rewriting methods continue to produce invalid or incorrect instances
- Key finding: Models exhibit accuracy drops on proof-verified rewritten benchmarks, indicating sensitivity to surface-level and structural variations
Industry Insight
- Benchmark contamination is a growing concern as LLMs are trained on increasingly large and comprehensive datasets; formal verification-based rewriting should become a standard practice for evaluation pipelines
- The accuracy drops observed on rewritten benchmarks suggest that many published LLM scores may overestimate genuine mathematical reasoning ability, warranting re-evaluation of model rankings using proof-verified benchmarks
- The integration of Lean ATPs into evaluation workflows opens a new paradigm where formal methods and neural approaches complement each other, potentially influencing how future benchmarks are designed and validated
Disclaimer: The above content is generated by AI and is for reference only.