Is RAG Dead? What My Own Numbers Say
Long-context models (gemini-3.1-flash-lite) achieved 96% accuracy on factual QA from a 145K-token corpus, proving modern models are genuinely capable at finding specific facts in large text piles RAG achieved 88% accuracy with top-5 chunk retrieval, showing both approaches are "good enough" on accuracy for single-fact questions at this scale Long-context cost 75x more per query than RAG due to resending the entire corpus on every request, making cost the primary differentiator Latency gap was mi
Analysis
TL;DR
- Long-context models (gemini-3.1-flash-lite) achieved 96% accuracy on factual QA from a 145K-token corpus, proving modern models are genuinely capable at finding specific facts in large text piles
- RAG achieved 88% accuracy with top-5 chunk retrieval, showing both approaches are "good enough" on accuracy for single-fact questions at this scale
- Long-context cost 75x more per query than RAG due to resending the entire corpus on every request, making cost the primary differentiator
- Latency gap was minimal (2.5s vs 3.0s), contradicting widely-cited claims of 20-60+ second differences for long-context
- RAG's role has shifted from compensating for weak readers to keeping strong readers affordable and viable as corpus size and usage scale
Why It Matters
This directly challenges the "RAG is dead" narrative gaining traction as context windows expand, providing empirical data rather than hot takes on when retrieval is actually necessary. For AI practitioners building production systems, the findings clarify that the decision between RAG and long-context should be driven by cost and scale considerations rather than accuracy concerns at current model capabilities.
Technical Details
- Corpus: 12 Wikipedia articles on space exploration (Apollo missions, ISS, Voyager, Mars rovers, Hubble, JWST, SpaceX) with 509 chunks at 1500 characters with 200-character overlap, generating 24 questions with known reference answers
- RAG pipeline: Plain baseline chunking, cosine similarity embedding retrieval of top-5 chunks, same gemini-3.1-flash-lite model for answer generation
- Long-context approach: Entire corpus (~145K tokens, ~14.5% of 1M token window) sent as context on every query with identical prompt template
- Scoring methodology: LLM-as-judge comparing generated answers against reference answers, with hand cross-checking of sampled judgments; retrieval tracked at both article-level and exact-chunk-level to distinguish retrieval failures from generation failures
- Cost calculation: Real token counts pulled from API response metadata using pricing of $0.25/M input and $1.50/M output tokens for gemini-3.1-flash-lite
Industry Insight
- Treat RAG as a cost-optimization and scaling strategy rather than an accuracy necessity; design your retrieval architecture based on expected query volume and corpus size growth, not on assumptions about model capability gaps
- Implement dual-level retrieval tracking (article-level and chunk-level) in your own systems to properly diagnose whether production RAG failures stem from the retriever or the generator, avoiding misdirected optimization efforts
- For corpora exceeding model context windows or requiring frequent document updates, RAG remains structurally necessary regardless of long-context accuracy gains; evaluate the total cost of ownership including re-indexing versus resending costs over time
Disclaimer: The above content is generated by AI and is for reference only.