ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning
ClosureBench is a constructive benchmark for compositional graph-relational reasoning with programmatically verified ground truth via the Ein tensor-logic language, generating instances on demand to eliminate data contamination concerns The benchmark spans 26 task categories across three compositional levels (L1-L3), with difficulty controlled along three independent axes: graph size, edge density, and query depth Fine-tuning on a fixed test set creates a 19.3 percentage-point memorization gap b
Analysis
TL;DR
- ClosureBench is a constructive benchmark for compositional graph-relational reasoning with programmatically verified ground truth via the Ein tensor-logic language, generating instances on demand to eliminate data contamination concerns
- The benchmark spans 26 task categories across three compositional levels (L1-L3), with difficulty controlled along three independent axes: graph size, edge density, and query depth
- Fine-tuning on a fixed test set creates a 19.3 percentage-point memorization gap between seen and fresh instances, a flaw invisible to static benchmarks
- Frontier models degrade from atomic to compositional queries because they misread graphs from natural-language descriptions and then reason correctly over the wrong graph—a bottleneck inherent to reasoning, not input format
- A 4B model fine-tuned to emit executable programs (Ein or Python+NetworkX) instead of direct answers stays nearly flat across compositional levels and reaches 94.3% accuracy on held-out instances at a fraction of the token cost
Why It Matters
This benchmark directly addresses the growing crisis of data contamination in AI evaluation by replacing static test sets with on-demand programmatic generation, enabling honest measurement of generalization versus memorization. It reveals a fundamental reasoning bottleneck in even frontier models when handling compositional graph tasks, suggesting that current architectures struggle with multi-step relational reasoning regardless of input modality. The program-synthesis approach demonstrates that smaller models can match frontier performance on complex reasoning tasks, offering a cost-effective path forward for production deployments.
Technical Details
- Benchmark design: ClosureBench generates instances on demand using the Ein tensor-logic language for ground-truth computation, ensuring machine-verified correctness. It covers 26 task categories at three compositional levels (L1 atomic, L2 two-step, L3 multi-step), with difficulty parameterized along graph size, edge density, and query depth.
- Evaluation scope: Models tested range from 1.5B open-weight models to frontier systems including o3, GPT-4.1, Gemini 2.5, and Claude Sonnet 4, providing a comprehensive comparison across the capability spectrum.
- Memorization measurement: By evaluating fine-tuned models on both seen and fresh instances, the benchmark quantifies a 19.3 percentage-point accuracy gap, isolating supervised fine-tuning on answer pairs from pretraining contamination.
- Input format robustness: The reasoning bottleneck persists across natural-language descriptions, JSON edge lists, and adjacency matrices, confirming the limitation is architectural rather than format-dependent.
- Program synthesis approach: A 4B model fine-tuned to emit executable programs in either Ein or Python+NetworkX achieves 94.3% accuracy on held-out instances with significantly reduced token costs, demonstrating that verified program synthesis generalizes across target languages.
Industry Insight
- Static benchmarks are becoming increasingly unreliable as model training data overlaps with popular evaluation sets; the industry should adopt constructive benchmarks with on-demand instance generation to maintain evaluation integrity.
- The finding that reasoning bottlenecks persist regardless of input format suggests that architectural improvements—not just prompt engineering or format optimization—are needed to advance compositional graph reasoning capabilities.
- Program synthesis fine-tuning of smaller models offers a compelling alternative to scaling frontier models for reasoning-intensive tasks, potentially reducing inference costs by orders of magnitude while maintaining competitive accuracy, making it a viable strategy for production-grade AI systems.
Disclaimer: The above content is generated by AI and is for reference only.