Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
NEEDLE is a live, open-source web search benchmark that continuously regenerates queries from fresh public sources (RSS feeds, SEC XBRL, arXiv, CourtListener, etc.) to prevent overfitting and answer-key leakage It evaluates 15 search APIs across five verticals—News, Everyday, Expert, Deep-tail, and Legal—using a unified protocol with identical query text, evidence caps, and sequential request handling Every engine's score is measured against "ultimate," a pooled oracle engine that combines resul
Analysis
TL;DR
- NEEDLE is a live, open-source web search benchmark that continuously regenerates queries from fresh public sources (RSS feeds, SEC XBRL, arXiv, CourtListener, etc.) to prevent overfitting and answer-key leakage
- It evaluates 15 search APIs across five verticals—News, Everyday, Expert, Deep-tail, and Legal—using a unified protocol with identical query text, evidence caps, and sequential request handling
- Every engine's score is measured against "ultimate," a pooled oracle engine that combines results from all providers to establish an empirical quality ceiling
- Finance search is near-solved (top engines at 0.87–0.91 vs. 0.965 ceiling), while deep-tail rare-entity queries remain the hardest challenge with the best engine at only 0.557 of ceiling
- The benchmark is fully reproducible: MIT-licensed Python CLI, runs on a laptop or CI, with public GitHub Actions and Hugging Face dataset artifacts
Why It Matters
NEEDLE directly addresses a critical flaw in AI evaluation—static benchmarks that agents can game by fetching gold labels or relying on parametric memory—by using dynamically regenerated queries from live sources. For AI practitioners building agentic search systems, it provides the first standardized way to distinguish whether poor results stem from ranking deficiencies or fundamental retrieval gaps shared across the entire market.
Technical Details
- Dynamic query generation: News queries regenerate hourly from ~124 RSS feeds and Google Trends; finance, scholar, legal, and deep-tail queries regenerate daily from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs (DeepResearchGym, OpenResearcher, LRAT)
- Five evaluation verticals: News (keyword queries from trending items), Finance (registry facts + 10-Q figures), Scholar (four query styles per paper: degraded title, full-text detail, natural-language clue, hedged description), Deep-tail (rare-word queries from real agent trajectories), Legal (CourtListener opinions across 14 federal courts)
- Scoring methodology: News and deep-tail use LLM judges (0–4 scale) with nDCG@5 and duplicate-URL penalty; Finance uses answer-recall@5; Scholar and Legal use known-item identifier matching
- Controlled evaluation protocol: One call at a time per engine, 2,000-character evidence clip, no page fetching or re-ranking, blind judging (judge doesn't see engine name), enabling comparable latency percentiles
- Ultimate ceiling: Pools all engines' results per query into a synthetic oracle, then ranks by relevance—creating an empirical upper bound that distinguishes ranking problems from retrieval problems
Industry Insight
- The widening gap between delivered and achievable quality on deep-tail queries (0.557 ceiling for top engine) signals that real-world agentic search remains significantly unsolved—invest and research should prioritize rare-entity and long-tail retrieval over well-structured domains like finance
- Latency disparities are massive (Keenable at 193ms p50 vs. Bing at 2,767ms p50), making speed a decisive factor for agents that make dozens of search calls per task; practitioners should weight latency-aware benchmarking alongside accuracy
- The "ultimate ceiling" framework offers a new evaluation paradigm: rather than absolute scores, the gap to pooled-oracle performance reveals whether an engine has headroom for improvement or whether the entire field faces a shared retrieval limitation—this distinction should guide whether to invest in ranking optimization versus foundational retrieval upgrades
Disclaimer: The above content is generated by AI and is for reference only.