Reduce RAG costs on Amazon Bedrock with query-aware compression
Query-aware compression uses a smaller, lower-cost model (Claude Haiku) to filter retrieved chunks against the user query before the primary model (Claude Sonnet) generates the final answer This post-retrieval pattern significantly reduces input tokens sent to the expensive primary model, yielding meaningful cost savings at scale The approach is compatible with Amazon Bedrock Knowledge Bases and existing RAG retrievers, requiring only a single additional compression call in the pipeline Removing
Analysis
TL;DR
- Query-aware compression uses a smaller, lower-cost model (Claude Haiku) to filter retrieved chunks against the user query before the primary model (Claude Sonnet) generates the final answer
- This post-retrieval pattern significantly reduces input tokens sent to the expensive primary model, yielding meaningful cost savings at scale
- The approach is compatible with Amazon Bedrock Knowledge Bases and existing RAG retrievers, requiring only a single additional compression call in the pipeline
- Removing irrelevant context not only cuts costs but also reduces the surface area for hallucinations by eliminating distracting information
- The pattern can be layered on top of existing Amazon Bedrock capabilities like prompt caching, Intelligent Prompt Routing, and the Rerank API for compounding cost savings
Why It Matters
This approach directly addresses one of the most significant cost drivers in production RAG systems: the volume of input tokens sent to foundation models. For AI practitioners building enterprise RAG applications at scale, the ability to maintain answer quality while substantially reducing per-query costs represents a practical optimization path that doesn't require architectural overhauls. The pattern's compatibility with existing Amazon Bedrock infrastructure makes it immediately actionable for teams already invested in the AWS ecosystem.
Technical Details
- Architecture: A two-stage model cascade where Claude Haiku (compression model) reads retrieved chunks alongside the user query and outputs only verbatim relevant spans, which are then fed to Claude Sonnet (primary model) for answer generation
- Implementation: Both the compression call and answer call run inside a single AWS Lambda function using the Amazon Bedrock Converse API, with the retriever (e.g., Amazon Bedrock Knowledge Bases backed by OpenSearch Serverless) returning top-k chunks upstream
- Economic model: Savings depend on the price ratio between small and primary models and the compression ratio achieved; cost formula accounts for R retrieved tokens, compression ratio c, output tokens A, and per-token prices for both models
- Integration points: The pattern is designed to compound with existing Bedrock features including prompt caching, Intelligent Prompt Routing, and the Rerank API for additional optimization
- Prerequisites: Requires an AWS account, IAM role with Bedrock model access for both Claude Haiku and Claude Sonnet in the target region
Industry Insight
- The query-aware compression pattern represents a practical "cascade" strategy that AI teams can deploy without migrating away from their current RAG infrastructure, making it a low-risk optimization for production systems
- As RAG workloads scale, the compounding effect of combining multiple optimization layers (compression + prompt caching + reranking + intelligent routing) will become essential for maintaining cost-effective operations
- The dual benefit of cost reduction and hallucination mitigation through context pruning suggests this pattern should become a standard component in enterprise RAG architectures, particularly for document-heavy domains like legal and technical documentation where retrieval typically returns high-recall but noisy context
Disclaimer: The above content is generated by AI and is for reference only.