Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
Current LLM query routers optimize for accuracy and cost but ignore latency, relying on naive load-balancing policies like round-robin. The authors introduce a lightweight latency estimator that simulates autoregressive token batch processing to predict Time-To-First-Token (TTFT). A new latency-aware router jointly optimizes latency, accuracy, and cost when assigning queries to model instances. Experimental results show up to a 40% improvement in accuracy-cost utility while maintaining latency l
Analysis
TL;DR
- Current LLM query routers optimize for accuracy and cost but ignore latency, relying on naive load-balancing policies like round-robin.
- The authors introduce a lightweight latency estimator that simulates autoregressive token batch processing to predict Time-To-First-Token (TTFT).
- A new latency-aware router jointly optimizes latency, accuracy, and cost when assigning queries to model instances.
- Experimental results show up to a 40% improvement in accuracy-cost utility while maintaining latency levels comparable to standard load-balancing approaches.
Why It Matters
This research addresses a critical gap in production LLM deployments where user experience is heavily influenced by response time, not just cost or quality. By integrating latency estimation into the routing decision process, practitioners can achieve better performance trade-offs without sacrificing speed, making it essential for real-time applications requiring high throughput and low delay.
Technical Details
- Problem Identification: Existing routers are latency-agnostic, treating latency as a secondary effect of load-balancing rather than a primary optimization variable dependent on prompt length, prefill/decode workload, and serving framework policies.
- Latency Estimator Design: A lightweight simulator is developed to model autoregressive token batch processing within the serving framework, specifically estimating the Time-To-First-Token (TTFT) for incoming queries.
- Joint Optimization Router: The estimator is integrated into a routing mechanism that simultaneously considers three metrics: latency, accuracy, and monetary cost, allowing for dynamic assignment based on current system state.
- Performance Metrics: The approach was evaluated against standard load-balancing policies, demonstrating significant gains in the accuracy-cost utility metric while keeping latency constraints satisfied.
Industry Insight
- Holistic Routing Strategies: AI infrastructure teams should move beyond simple cost/accuracy trade-offs and implement latency-aware routing to enhance user satisfaction, particularly in interactive applications.
- Serving Framework Integration: Effective routing requires deep integration with the underlying serving engine’s batching and scheduling policies; abstracting away these details limits optimization potential.
- Utility Maximization: The 40% improvement in accuracy-cost utility suggests that ignoring latency leads to suboptimal resource allocation, and accounting for it can significantly reduce operational costs for the same quality of service.
Disclaimer: The above content is generated by AI and is for reference only.