ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence
ESQ-Bench is introduced as an Oracle-first NL2SQL benchmark addressing the gap between academic benchmarks (Spider, BIRD) and real-world enterprise database complexity The benchmark features six populated schemas across four SQL dialects (Oracle, PostgreSQL, MySQL, SQL Server) with 465 tables, 164,682 rows, and zero empty tables, organized into three complexity tiers A four-metric evaluation harness (Exact Match, Execution Accuracy, Semantic Recall, Silent Divergence) reveals that operational si
Analysis
TL;DR
- ESQ-Bench is introduced as an Oracle-first NL2SQL benchmark addressing the gap between academic benchmarks (Spider, BIRD) and real-world enterprise database complexity
- The benchmark features six populated schemas across four SQL dialects (Oracle, PostgreSQL, MySQL, SQL Server) with 465 tables, 164,682 rows, and zero empty tables, organized into three complexity tiers
- A four-metric evaluation harness (Exact Match, Execution Accuracy, Semantic Recall, Silent Divergence) reveals that operational silent-divergence reaches 73-99% among execution-passing queries, exposing a critical evaluation blind spot
- Claude Sonnet 4.6 outperforms GPT-4o across all tiers with schema-linked prompting (87.4%, 74.9%, 68.7% EX), while local Llama 3.2 achieves only 13.3% bank-wide EX
- GPT-4o zero-shot performance inverts schema-linked results at Tiers 2-3 due to survivor bias and lower execution rates, highlighting methodological pitfalls in benchmark evaluation
Why It Matters
This benchmark exposes a fundamental flaw in NL2SQL evaluation: high execution accuracy on academic datasets masks catastrophic silent semantic divergence in enterprise settings, where queries may execute successfully but return wrong results. For AI practitioners building production NL2SQL systems, this work provides the first systematic framework to detect and quantify silent semantic failures that traditional metrics completely miss.
Technical Details
- Schema Construction: Six enterprise-grade schemas deployed identically across Oracle, PostgreSQL, MySQL, and SQL Server with 465 tables, 164,682 rows, and zero empty tables, ensuring cross-dialect comparability
- Three-Tier Complexity Structure: Tier-1 (95 questions), Tier-2 (228 questions), Tier-3 (227 questions) with systematically increasing schema complexity and query difficulty
- Four-Metric Evaluation Harness: Exact Match (EM), Execution Accuracy (EX), Semantic Recall (SR), and Silent Divergence (SD) — with SD specifically designed to detect cases where queries execute without error but produce incorrect results
- Model Evaluation Results: GPT-4o schema-linked EX degrades monotonically across tiers (79.8%, 60.3%, 57.2%); Claude Sonnet 4.6 achieves 87.4%, 74.9%, 68.7%; Llama 3.2 reaches only 13.3% bank-wide (73/550)
- Critical Finding: EM remains below 7% tier-wide, and wrong-result semantics dominate failure modes at higher tiers, with silent-divergence rates of 73-99% among EX-passing queries
Industry Insight
- Organizations deploying NL2SQL in enterprise environments must adopt silent-divergence-aware evaluation; relying solely on execution accuracy will produce systems that appear functional but deliver incorrect business results
- The dramatic performance gap between Claude Sonnet 4.6 and GPT-4o on enterprise Oracle schemas suggests dialect-specific fine-tuning and schema-aware prompting strategies are critical for production deployments
- The survivor bias phenomenon observed in GPT-4o zero-shot vs. schema-linked comparisons serves as a cautionary tale: benchmark methodology must account for execution rate differences to avoid misleading performance rankings
Disclaimer: The above content is generated by AI and is for reference only.