Benchmarking Hybrid Deep Research Across Database Querying and Web Search
HybridDeepResearch is introduced as the first deep-research benchmark requiring both web search and SQL querying to form complete, verifiable answers, addressing the gap in existing benchmarks that evaluate these modalities in isolation The benchmark contains 380 tool-dependent tasks grounded in LiveSQLBench-Base-Lite databases and public web corpora, validated through automated checks and human review Three reasoning patterns are defined: SQL2S (database-to-web), S2SQL (web-to-database), and Pa
Analysis
TL;DR
- HybridDeepResearch is introduced as the first deep-research benchmark requiring both web search and SQL querying to form complete, verifiable answers, addressing the gap in existing benchmarks that evaluate these modalities in isolation
- The benchmark contains 380 tool-dependent tasks grounded in LiveSQLBench-Base-Lite databases and public web corpora, validated through automated checks and human review
- Three reasoning patterns are defined: SQL2S (database-to-web), S2SQL (web-to-database), and Parallel (simultaneous intersection of both sources)
- State-of-the-art models (GLM-5.2, Claude-Sonnet-4.6, GPT-5) achieve only 50-54% Pass@8 on the hard subset, revealing significant performance gaps
- Directional reasoning (SQL2S and S2SQL) proves substantially more difficult than parallel intersection, highlighting the critical "handoff" challenge of preserving constraints when moving evidence between structured and unstructured information spaces
Why It Matters
This benchmark directly addresses a fundamental limitation in current AI agent evaluation: real-world analytical tasks require agents to seamlessly integrate evidence from both unstructured web sources and structured databases, yet no existing benchmark measures this critical capability. For AI practitioners building agentic systems, the findings reveal that even top-tier models struggle to maintain constraint fidelity during cross-modal reasoning, signaling that hybrid research agents remain far from production readiness for complex enterprise queries.
Technical Details
- Benchmark composition: 380 tool-dependent tasks drawn from LiveSQLBench-Base-Lite databases paired with public web corpora, with validation through both automated checks and human review
- Reasoning patterns: SQL2S requires starting from database constraints and refining with web search; S2SQL inverts this flow by beginning with web findings and grounding them in SQL queries; Parallel requires simultaneous, intersecting use of both modalities
- Evaluation metric: Pass@8 measures whether at least one of eight generated trajectories produces a correct, verifiable answer
- Models evaluated: Proprietary (GLM-5.2, Claude-Sonnet-4.6, GPT-5) and open-weight models tested under various agentic scaffolds
- Code and data: Publicly available on GitHub and Hugging Face for reproducibility
Industry Insight
- The 50-54% Pass@8 ceiling for frontier models on hard tasks indicates that hybrid deep research agents will require significant architectural improvements—particularly in constraint preservation during cross-modal handoffs—before they can be reliably deployed for enterprise-grade analytical workflows
- The asymmetry between directional and parallel reasoning suggests that agent scaffolds should prioritize bidirectional tool orchestration rather than unidirectional pipelines, as the web-to-database and database-to-web transitions represent distinct failure modes that need targeted optimization
- Benchmark designers and practitioners should treat the SQL2S/S2SQL/Parallel taxonomy as a diagnostic framework for evaluating agent weaknesses, enabling more granular improvement roadmaps rather than relying on monolithic accuracy scores
Disclaimer: The above content is generated by AI and is for reference only.