LAI #139: Fewer Tokens Cost Us More
Chat history is often not the primary context bottleneck; old retrieval results and tool outputs can dominate token usage, contributing up to 100k tokens per turn Prompt caching economics can make summarization counterproductive: keeping full history retained ~87% cache discount while compaction reduced tokens by 41% but doubled cost Capping tool outputs at a stable size reduced cost per turn by 38% with no measurable quality loss, outperforming summarization-based approaches Hybrid retrieval (B
Analysis
TL;DR
- Chat history is often not the primary context bottleneck; old retrieval results and tool outputs can dominate token usage, contributing up to 100k tokens per turn
- Prompt caching economics can make summarization counterproductive: keeping full history retained ~87% cache discount while compaction reduced tokens by 41% but doubled cost
- Capping tool outputs at a stable size reduced cost per turn by 38% with no measurable quality loss, outperforming summarization-based approaches
- Hybrid retrieval (BM25 + dense) is essential: dense retrieval dropped to 0% recall at 400k-token corpora while BM25 maintained 100%, and GraphRAG cost 44% more per turn with no quality gain
- The core principle: name your constraint first (window size, cost, or quality degradation), as each requires a fundamentally different optimization strategy
Why It Matters
This article provides empirically grounded, production-tested guidance on context engineering that directly challenges common assumptions—particularly the instinct to summarize or compress conversation history. For AI practitioners building agentic systems, the findings offer a roadmap for reducing costs without sacrificing capability, while highlighting retrieval fusion strategies that many teams overlook. The $590 eval experiment serves as a cautionary template for how systematic benchmarking can expose flawed defaults before they scale.
Technical Details
- Context composition analysis: In a production AI tutor, retrieval outputs alone contributed up to 100k tokens per turn, with some turns reaching ~200k tokens—far exceeding chat history as the primary context consumer
- Cache economics: Full history retention achieved ~87% cached input discount at under $0.55 per million tokens; compaction reduced tokens by 41% but lost cache continuity, resulting in roughly 2x the cost
- Tool output capping: Implementing stable size caps on tool outputs cut cost per turn by 38% with no measured degradation in memory or response quality, outperforming summarization-based compaction
- Retrieval performance: Dense retrieval failed completely (0% recall) at 400k-token corpora in buried-fact tests, while BM25 maintained 100% recall; GraphRAG achieved equal accuracy but consumed 61% more tokens and cost 44% more per turn
- Hybrid search fusion strategy: Early merging of semantic and keyword results can eliminate exact matches; the fix is to preserve top-5 candidates from each retriever before merging and deduplicating
- Production stack: DeepSeek V4 Flash, hybrid retrieval, full history retention, stable tool output caps, no tool-output clearing, and summarization deferred as a last resort
Industry Insight
- Organizations should audit their context composition before investing in summarization or compaction—most teams are optimizing the wrong component. Retrieval and tool output management typically offer higher ROI than history compression.
- Prompt caching should be treated as a first-class economic factor in agent design; strategies that preserve cache continuity (like stable output capping) can dramatically outperform those that rewrite cached prefixes, even if they use more raw tokens.
- Hybrid retrieval with careful fusion ordering is a low-effort, high-impact improvement. Teams relying on semantic search alone risk catastrophic recall degradation at scale, while simple keyword retrieval remains robust—preserving both streams in the fusion pipeline is critical.
Disclaimer: The above content is generated by AI and is for reference only.