Perplexity AI Releases WANDR: An Open Benchmark Evaluating Research Agents That Must Search Wide And Deep
Perplexity AI introduces WANDR, an open benchmark designed to evaluate "wide and deep" research agents that must discover large sets of entities and support each with specific evidence. The benchmark utilizes a composable qualification key hierarchy to validate complex, multi-level data collection tasks, moving beyond simple single-answer accuracy. Evaluation metrics include Precision, Recall, and F1 scores calculated via a rigorous grading process that re-fetches URLs and verifies excerpt valid
Analysis
TL;DR
- Perplexity AI introduces WANDR, an open benchmark designed to evaluate "wide and deep" research agents that must discover large sets of entities and support each with specific evidence.
- The benchmark utilizes a composable qualification key hierarchy to validate complex, multi-level data collection tasks, moving beyond simple single-answer accuracy.
- Evaluation metrics include Precision, Recall, and F1 scores calculated via a rigorous grading process that re-fetches URLs and verifies excerpt validity against claims.
- Initial results show significant performance gaps, with Perplexity's Search as Code leading but achieving only moderate success rates, highlighting current limitations in agentic reliability.
- Key findings indicate that discovery bottlenecks and the difficulty of extracting precise evidence from retrieved pages are major challenges for existing AI systems.
Why It Matters
This benchmark addresses a critical gap in AI evaluation by testing agents on complex, real-world knowledge work tasks rather than isolated Q&A scenarios. It provides a standardized way to measure the reliability and completeness of research agents, which is essential for enterprises deploying AI for due diligence, competitive intelligence, and literature reviews. The results offer actionable insights into where current models fail, guiding future development in agentic workflows and retrieval-augmented generation.
Technical Details
- Benchmark Structure: WANDR consists of 500 realistic tasks derived from de-identified production patterns, requiring agents to build large, evidence-backed collections. Tasks are structured using a composable qualification key hierarchy (e.g.,
company(n) -> employee(m) -> url(k)). - Evaluation Methodology: The grader re-fetches submitted URLs to verify usability and scope. It checks if selected excerpts actually appear on the page and if they fully support the claimed answer. Metrics roll up from binary record verdicts to hierarchical Precision, Recall, and F1 scores.
- Task Complexity: Tasks vary in difficulty based on per-record work rather than just scale. The median task requires 50 members and 245 records, with the total benchmark demanding over 170,000 source-backed records across 500 tasks.
- Performance Analysis: Perplexity's Search as Code (SaC) system achieved the highest Soft F1 of 0.363 and Hard F1 of 0.133. Other systems like Anthropic showed lower scores despite higher costs, while others were faster/cheaper but less accurate.
- Key Bottlenecks: Analysis reveals that top-level discovery completion varies significantly (0.611 to 0.951), and turning a usable page into complete evidence is difficult, with over 57% of excerpts failing to support the full claim in some cases.
Industry Insight
- Shift from Generation to Verification: As AI agents move into professional knowledge work, evaluation must prioritize evidence verification and completeness over mere narrative generation. Benchmarks like WANDR set a new standard for assessing agent reliability in high-stakes environments.
- Architectural Implications: The success of "Search as Code" suggests that deterministic programming structures for retrieval, filtering, and fan-out logic are superior to purely generative approaches for wide-and-deep tasks. Developers should consider hybrid architectures that separate discovery logic from LLM reasoning.
- Cost-Performance Trade-offs: The wide range of costs ($0.03 to $324.83 per task) highlights the inefficiency of current agentic workflows. Organizations must balance token usage and latency against the need for exhaustive evidence, potentially requiring specialized optimization for large-scale data collection.
Disclaimer: The above content is generated by AI and is for reference only.