Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting
RAG systems exhibit a shifting bottleneck between upstream reranking and downstream LLM generation depending on query rates and reranking budgets, challenging the assumption that downstream optimization is always the priority PACE (Prioritized Adaptive Coverage of Evidence) is a training-free framework combining evidence frontloading and pressure-adaptive budgeting to address this shifting bottleneck Evidence frontloading reorders candidates by marginal evidence coverage, prioritizing query-rele
Analysis
TL;DR
- RAG systems exhibit a shifting bottleneck between upstream reranking and downstream LLM generation depending on query rates and reranking budgets, challenging the assumption that downstream optimization is always the priority
- PACE (Prioritized Adaptive Coverage of Evidence) is a training-free framework combining evidence frontloading and pressure-adaptive budgeting to address this shifting bottleneck
- Evidence frontloading reorders candidates by marginal evidence coverage, prioritizing query-relevant, complementary documents that form multi-hop evidence chains, with a monotone submodular objective guaranteeing a (1-1/e) approximation via greedy selection
- Pressure-adaptive budgeting dynamically adjusts the reranking budget based on the relative computational pressure between the reranker and the LLM
- Experiments on three multi-hop QA datasets and online serving simulations demonstrate improved evidence recall and reduced p95 latency under ranking-heavy workloads, validating the "less can be more" principle
Why It Matters
This work fundamentally reframes how practitioners approach RAG efficiency by demonstrating that the bottleneck is not fixed but shifts across the pipeline, meaning optimization strategies must be workload-aware rather than one-size-fits-all. For AI engineers deploying RAG systems at scale, PACE offers a practical, training-free solution that can be integrated without retraining models, directly addressing latency and recall trade-offs that are critical for production systems. The theoretical guarantee on the submodular objective also provides a principled foundation for evidence selection that goes beyond heuristic approaches.
Technical Details
- Shifting-bottleneck characterization: The paper empirically demonstrates that under high query rates or large reranking budgets, upstream reranking becomes the dominant bottleneck rather than downstream LLM generation, overturning the common assumption that RAG efficiency efforts should focus primarily on the generator
- Evidence frontloading: PACE reorders retrieved candidates by marginal evidence coverage, selecting documents that are query-relevant, complementary to already-selected documents, and useful for constructing multi-hop evidence chains; the objective function is proven to be monotone submodular, enabling greedy selection with a (1-1/e) approximation guarantee
- Pressure-adaptive budgeting: The framework dynamically allocates the reranking budget based on real-time relative pressure between the reranker and the LLM, reducing the budget when reranking is the bottleneck while preserving enough evidence for recall
- Evaluation: Tested on three multi-hop QA datasets with online serving simulations, showing improved evidence recall and reduced p95 latency under ranking-heavy workloads, with the combined components demonstrating that fewer but more evidence-dense top-ranked candidates yield higher final recall
Industry Insight
- Organizations running RAG at scale should monitor where bottlenecks actually sit in their pipeline rather than defaulting to downstream optimization; workload-aware budgeting can yield significant latency improvements without model changes
- The training-free nature of PACE makes it immediately deployable in existing RAG infrastructures, offering a low-risk, high-impact optimization path for production systems facing high query throughput
- The "less can be more" insight suggests that investing in smarter evidence selection at the reranking stage can be more effective than simply increasing context window size or reranking budget, pointing toward a design philosophy prioritizing evidence density over evidence quantity
Disclaimer: The above content is generated by AI and is for reference only.