SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs
SearchAtlas is a framework that converts LLM search agent trajectories into structured evidential query graphs, mapping how evidence propagates from queries to final answers The automated parsing pipeline achieves 86.0% mean edge F1 against human-annotated graphs with consistent performance across repeated runs Analysis of five search agents across three benchmarks reveals systematic differences in search scale and evidence aggregation patterns SearchAtlas exposes three key process failures: fra
Analysis
TL;DR
- SearchAtlas is a framework that converts LLM search agent trajectories into structured evidential query graphs, mapping how evidence propagates from queries to final answers
- The automated parsing pipeline achieves 86.0% mean edge F1 against human-annotated graphs with consistent performance across repeated runs
- Analysis of five search agents across three benchmarks reveals systematic differences in search scale and evidence aggregation patterns
- SearchAtlas exposes three key process failures: fragmented answer support, unreachable question constraints, and unverified parametric knowledge entering responses
- Process-diagnostic scores from the graph analysis correlate more strongly with answer correctness than LLM judges evaluating raw trajectories or ordered query lists
Why It Matters
This work addresses a critical gap in LLM agent evaluation: the overreliance on final-answer accuracy while ignoring the reasoning process. By making search strategies interpretable through graph structures, SearchAtlas gives practitioners a diagnostic tool to identify exactly where and why agents fail, enabling targeted improvements rather than blind benchmarking.
Technical Details
- Graph Construction: SearchAtlas transforms raw search trajectories into structured graphs where nodes represent queries and evidence, and edges encode the propagation of evidence from retrieval to final answer generation
- Parsing Pipeline: An automated pipeline parses agent trajectories into these graphs, achieving 86.0% mean edge F1 against human-annotated ground truth, with consistency across repeated runs
- Evaluation Scope: Five distinct search agents were analyzed across three benchmarks, examining differences in search scale (number of queries) and evidence aggregation strategies
- Diagnostic Metrics: The framework quantifies process failures including fragmented answer support (evidence not cohesively linked), question constraint coverage (whether all constraints are addressed), and parametric knowledge leakage (unverified internal knowledge used as evidence)
- Comparative Analysis: Process-diagnostic scores were compared against LLM judge evaluations on both raw trajectories and ordered query lists, demonstrating superior correlation with actual answer correctness
Industry Insight
- Agent evaluation frameworks should incorporate process-level diagnostics alongside outcome metrics; final accuracy alone masks systematic reasoning failures that prevent generalization
- The graph-based interpretability approach could be adapted for other agentic workflows beyond search, such as code generation or multi-step planning, where traceability of evidence is equally critical
- Teams building production search agents should monitor evidence propagation completeness and constraint coverage as early-warning signals, since process failures strongly predict answer errors even when the final answer appears correct
Disclaimer: The above content is generated by AI and is for reference only.