Half the Answer Keys in Text-to-SQL Benchmarks Are Wrong.
A VLDB 2026 audit revealed that over 50% of answer keys in major Text-to-SQL benchmarks (BIRD, Spider 2.0) are incorrect due to structural flaws in manual annotation. The author proposes an inverse data generation method where ground-truth specifications are declared first, and databases are algorithmically generated to satisfy those constraints exactly. Verification is enforced via independent execution of gold SQL against generated CSVs using DuckDB, ensuring deterministic and reproducible cor
Analysis
TL;DR
- A VLDB 2026 audit revealed that over 50% of answer keys in major Text-to-SQL benchmarks (BIRD, Spider 2.0) are incorrect due to structural flaws in manual annotation.
- The author proposes an inverse data generation method where ground-truth specifications are declared first, and databases are algorithmically generated to satisfy those constraints exactly.
- Verification is enforced via independent execution of gold SQL against generated CSVs using DuckDB, ensuring deterministic and reproducible correctness without relying on human adjudication.
- This approach eliminates semantic ambiguities and schema misunderstandings by deriving questions directly from precise, pre-defined mathematical specifications.
Why It Matters
This development challenges the validity of current Text-to-SQL evaluation metrics, suggesting that many "state-of-the-art" models may be performing well on noisy or incorrect benchmarks rather than demonstrating true capability. For researchers and practitioners, it highlights the urgent need for verifiable, synthetic evaluation suites that guarantee answer key integrity, moving away from fragile human-annotated datasets toward mathematically grounded synthetic data pipelines.
Technical Details
- Inverse Construction Method: Instead of annotating existing databases, the system starts with explicit specifications (e.g., exact monthly revenue, fraud rates) and uses a deterministic generator to create databases that satisfy these constraints, employing lognormal distributions and rescaling algorithms.
- Independent Verification Protocol: Gold SQL queries are executed against the generated CSV files using DuckDB, a separate engine with no shared codebase with the generator, to independently verify that observed answers match declared targets exactly.
- Precision Handling: The method ensures float serialization round-trips correctly, preserving cent-exact sums from generation to file storage, thereby eliminating floating-point discrepancies common in traditional evaluations.
- Ambiguity Elimination: Questions are authored from the specification rather than inferred from data, explicitly defining time windows (e.g., half-open intervals) and output formats to prevent the semantic mismatches identified in the UIUC audit.
Industry Insight
- Benchmark Integrity Crisis: The high error rate in existing benchmarks implies that current leaderboards are unreliable indicators of model performance; organizations should prioritize models validated against synthetic, verifiable datasets over those ranked on traditional benchmarks.
- Shift to Synthetic Evaluation: The industry is likely to see a rapid adoption of synthetic data generation tools for evaluation purposes, where correctness is guaranteed by construction rather than assumed through human annotation.
- Standardization of Verification Artifacts: Future benchmarks may require accompanying verification scripts and cryptographic certificates (recording seeds, specs, and versions) to allow independent reproducibility, similar to the open-source pack demonstrated here.
Disclaimer: The above content is generated by AI and is for reference only.