Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents
Search APIs function as "decision surfaces" where snippet quality and ranking dictate agent behavior, rather than just serving as raw retrieval sources. Providers like Brave, Tavily, and Firecrawl yield nearly identical final answer accuracy (25-26%) but differ significantly in token efficiency and evidence distribution. The study introduces the "surface contradiction-to-gold URL ratio" to quantify how misleading search results can be before the agent even fetches the page. Agent retrieval strat
Analysis
TL;DR
- Search APIs function as "decision surfaces" where snippet quality and ranking dictate agent behavior, rather than just serving as raw retrieval sources.
- Providers like Brave, Tavily, and Firecrawl yield nearly identical final answer accuracy (25-26%) but differ significantly in token efficiency and evidence distribution.
- The study introduces the "surface contradiction-to-gold URL ratio" to quantify how misleading search results can be before the agent even fetches the page.
- Agent retrieval strategies should optimize for "evidence economy" (token cost vs. information gain) rather than focusing solely on final answer correctness.
Why It Matters
This research shifts the evaluation paradigm for agentic workflows from simple accuracy metrics to cost-efficiency and decision-making quality. For AI practitioners, it highlights that choosing a search provider is a strategic budgeting decision that impacts latency, token usage, and reliability, not just the final output. Understanding these "decision surfaces" allows developers to design more efficient retrieval-augmented generation (RAG) pipelines that minimize unnecessary page fetches.
Technical Details
- Experimental Setup: Evaluated a frozen GPT-5.4 agent using two tools (
search_webandfetch_page) against 100 questions from the SEALQA-HARD dataset. - Providers Tested: Compared three major search APIs: Brave, Tavily, and Firecrawl.
- Evaluation Metrics: Used a Kimi-K2.6 oracle to label 6,869 per-URL judgments, including a novel "surface contradiction-to-gold URL ratio" ranging from 0.92 to 2.59 across providers.
- Key Findings: While final accuracy was comparable (Brave: 25, Tavily: 25, Firecrawl: 26), Brave provided richer snippets, Tavily concentrated high-quality URLs at rank 1, and Firecrawl prompted broader exploration, affecting token consumption.
Industry Insight
- Optimize for Token Economy: When building agents, prioritize search providers that offer high-information-density snippets to reduce the need for expensive full-page fetches.
- Monitor Decision Surfaces: Implement monitoring for the "contradiction-to-gold" ratio to detect when search results are actively misleading the agent, even if final accuracy remains stable.
- Provider Selection Strategy: Treat search API selection as a policy decision based on specific agent behaviors (e.g., aggressive exploration vs. conservative fetching) rather than a generic benchmark score.
Disclaimer: The above content is generated by AI and is for reference only.