Round-Robin Broke My Agent
Round-robin load balancing across multiple LLM resources fails for stateful APIs like the OpenAI Responses API because server-side persisted items (with opaque IDs) must be replayed to the same resource that created them A 400 error carrying a "completed" response body is a diagnostic red flag indicating a routing bug, not a transient failure, often caused by gateway response-body capture logic Retry policies that retry all 4xx errors across a backend pool can waste up to 30x resources by accide
Analysis
TL;DR
- Round-robin load balancing across multiple LLM resources fails for stateful APIs like the OpenAI Responses API because server-side persisted items (with opaque IDs) must be replayed to the same resource that created them
- A 400 error carrying a "completed" response body is a diagnostic red flag indicating a routing bug, not a transient failure, often caused by gateway response-body capture logic
- Retry policies that retry all 4xx errors across a backend pool can waste up to 30x resources by accidentally performing resource-affinity searches at high cost
- Three viable fixes exist: disable server-side storage for stateless calls, implement consistent-hash session affinity for stateful workloads, or segregate stateful traffic to dedicated resources
Why It Matters
This reveals a critical gap between traditional HTTP load balancing assumptions and the realities of modern LLM APIs, where statefulness is increasingly common. For AI practitioners building production systems, blindly applying round-robin patterns across multiple provider endpoints can silently break agent workflows, inflate costs, and produce misleading error signals that are difficult to diagnose.
Technical Details
- The OpenAI Responses API with
store=truepersists every model output item server-side with resource-scoped IDs (fc_,rs_,msg_,resp_prefixes); replaying these IDs to a different Azure OpenAI resource returns a deterministic 400 with the message "The requested item was created under a different Azure OpenAI resource" - Prompt caching is also resource-scoped: round-robin distribution across N backends reduces warm-cache hit rates to roughly 1/N, silently increasing latency and cost without any correctness failure
- The diagnostic fingerprint for this class of bug is a binomial attempt pattern: N-1 failures followed by 1 success, with failures evenly distributed across all backends, detectable via telemetry queries grouping by operation and attempt shape
- Gateway response-body truncation at 8192 characters can mask the actual error, attributing a successful response from the retrying host to all failed attempts in the group, creating misleading logs
- Retry policies conditioned on
StatusCode >= 400retry every client error indiscriminately; the correct approach limits retries to 429/5xx/408 and caps the retry count to the number of distinct backends minus one
Industry Insight
- Load balancer architectures for LLM traffic must explicitly account for API statefulness; round-robin is only safe for pure stateless endpoints, and any API returning opaque IDs for subsequent turns requires affinity-based routing or stateless mode
- Pool homogeneity is non-negotiable: independently provisioned LLM resources drift in model versions, content-filter policies, and quota classes, turning a load balancer into a behavioral randomizer—automated drift detection and circuit breakers should be standard
- Retry budget design should be bounded by the number of useful backends, not optimism; pairing
first-fast-retry=truewith a count equal to the pool size ensures cross-backend retries explore new hosts immediately rather than burning time on already-refused ones
Disclaimer: The above content is generated by AI and is for reference only.