FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
Introduction of FindStatBench, an execution-based benchmark for evaluating LLMs on combinatorial code synthesis, featuring 2,329 tasks and 5.52M hidden instances. Evaluation of eleven systems reveals that top open- and closed-source models converge within 1 percentage point of instance accuracy. Counterintuitive finding that providing few-shot examples can degrade performance compared to zero-shot prompts for certain classical bijections. Significant challenges identified in map synthesis versus
Analysis
TL;DR
- Introduction of FindStatBench, an execution-based benchmark for evaluating LLMs on combinatorial code synthesis, featuring 2,329 tasks and 5.52M hidden instances.
- Evaluation of eleven systems reveals that top open- and closed-source models converge within 1 percentage point of instance accuracy.
- Counterintuitive finding that providing few-shot examples can degrade performance compared to zero-shot prompts for certain classical bijections.
- Significant challenges identified in map synthesis versus statistic synthesis, with long prompts causing sharp accuracy cliffs and output-budget constraints limiting code generation.
Why It Matters
This benchmark provides a rigorous, execution-based standard for assessing LLM capabilities in complex mathematical code generation, moving beyond static text evaluation. It highlights critical failure modes such as example-induced degradation and context window limitations, offering essential insights for developing more robust reasoning and coding agents.
Technical Details
- Benchmark Composition: Built on FindStat, containing 2,329 tasks across 24 collections, divided into statistic synthesis (mapping objects to integers) and map synthesis (mapping objects to objects).
- Evaluation Protocol: Models must generate a single Python
solvefunction without retrieval, tool use, or feedback loops; scoring is based on exact sandboxed execution against held-out combinatorial objects. - Model Scope: Assessed eleven systems, including four closed-source production models and seven open-weight models served via a single inference provider.
- Key Findings: Oracle aggregation and five-way sampling yielded limited gains, indicating diminishing returns from simple ensemble methods; exact symbolic rule induction remains brittle.
Industry Insight
- Developers should prioritize optimizing context window efficiency and output budget management, as long prompts significantly degrade performance in complex synthesis tasks.
- The counterintuitive negative impact of few-shot examples suggests that prompt engineering strategies may need revision, favoring zero-shot or carefully curated examples for specific mathematical domains.
- The convergence between open and closed-source models indicates that architectural differences may matter less than fine-tuning quality and inference protocols for this specific class of problems.
Disclaimer: The above content is generated by AI and is for reference only.