Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry
Introduces a new open-source benchmark isolating diagrammatic reasoning as a distinct capability from mathematical problem-solving in olympiad geometry The benchmark comprises 954 self-contained geometry problems (297 hard subset), each paired with a solution and a human-authored, high-fidelity diagram in renderable Asymptote code A multi-modal evaluation suite includes text-, code-, image-, VLM-, and constraint-based metrics to assess diagram construction fidelity Current foundation models exhi
Analysis
TL;DR
- Introduces a new open-source benchmark isolating diagrammatic reasoning as a distinct capability from mathematical problem-solving in olympiad geometry
- The benchmark comprises 954 self-contained geometry problems (297 hard subset), each paired with a solution and a human-authored, high-fidelity diagram in renderable Asymptote code
- A multi-modal evaluation suite includes text-, code-, image-, VLM-, and constraint-based metrics to assess diagram construction fidelity
- Current foundation models exhibit a pronounced solving-vs-drawing gap, achieving only a 36.14% average compile success rate on diagram generation
- Strong mathematical reasoning ability does not imply the ability to construct accurate geometric diagrams, revealing an unmeasured capability gap
Why It Matters
This benchmark addresses a critical blind spot in AI evaluation: while models like GPT and Claude excel at olympiad-level math, their ability to produce faithful geometric diagrams remains unmeasured and likely weak. For AI practitioners building systems that require spatial reasoning or multi-modal output, this work provides both a diagnostic tool and a clear target for improvement. The finding that mathematical reasoning and diagram construction are dissociable capabilities challenges the assumption that progress on one implies progress on the other.
Technical Details
- Dataset: 954 self-contained olympiad geometry problems with a 297-problem hard subset, each accompanied by a human-authored solution and a high-fidelity diagram encoded in renderable Asymptote code
- Evaluation Metrics: A multi-dimensional metric suite spanning text-based correctness, code-level compilation success, image fidelity, VLM-based assessment, and geometric constraint satisfaction
- Key Finding: Foundation models achieve an average compile success rate of only 36.14% on diagram generation, demonstrating a significant gap between symbolic reasoning and visual-spatial construction
- Open Source: The benchmark and dataset are publicly released for community evaluation and further research
Industry Insight
- Benchmark designers and model evaluators should incorporate diagrammatic reasoning assessments alongside traditional answer-based metrics to obtain a more complete picture of model capabilities in spatial-mathematical tasks
- Teams developing AI systems for education, CAD, or technical illustration should treat diagram construction as a separate capability requiring dedicated training and evaluation, rather than assuming it follows from reasoning proficiency
- The 36.14% compile success rate suggests that current foundation models lack robust internal geometric simulation, pointing to opportunities for improvement through hybrid architectures combining symbolic reasoning with explicit spatial representation modules
Disclaimer: The above content is generated by AI and is for reference only.