Why vLLM and SGLang Are Replacing Ollama for Agentic Workflows
Agent loops repeatedly re-send nearly identical prompts across hundreds of steps, creating significant redundant computation vLLM and SGLang implement persistent prompt caching that retains context across loop iterations, dramatically reducing redundant token processing Ollama drops cached prompts after five steps, forcing re-computation and introducing substantial latency overhead in long-running agent workflows Prompt caching efficiency is a critical differentiator for frameworks targeting age
Analysis
TL;DR
- Agent loops repeatedly re-send nearly identical prompts across hundreds of steps, creating significant redundant computation
- vLLM and SGLang implement persistent prompt caching that retains context across loop iterations, dramatically reducing redundant token processing
- Ollama drops cached prompts after five steps, forcing re-computation and introducing substantial latency overhead in long-running agent workflows
- Prompt caching efficiency is a critical differentiator for frameworks targeting agentic and multi-step reasoning applications
Why It Matters
For AI practitioners building agent-based systems, the choice of inference framework directly impacts performance and cost at scale. Agent loops are increasingly common in production RAG pipelines, autonomous agents, and multi-step reasoning workflows, making prompt caching a practical concern rather than a theoretical optimization. Frameworks that fail to maintain cache across iterations can incur orders-of-magnitude more compute than necessary.
Technical Details
- Prompt caching mechanism: In agent loops, the system prompt and prior conversation history remain largely static while only new tool outputs or responses change. Caching avoids re-processing the KV cache for unchanged prefix tokens.
- vLLM and SGLang: Both frameworks support continuous batching with prefix caching, maintaining the KV cache across successive requests within an agent loop. This enables near-zero overhead for repeated prompt prefixes.
- Ollama: Appears to clear or expire its prompt cache after approximately five inference steps, requiring full re-computation of the KV cache for the shared prompt prefix on subsequent calls.
- Performance impact: In loops running hundreds of steps, the difference between persistent caching and no caching translates to dramatically higher latency, GPU utilization, and inference costs with uncached frameworks.
Industry Insight
- Framework selection should prioritize caching behavior when designing agent systems; benchmarking prompt cache hit rates under realistic loop conditions is essential before committing to an inference backend.
- As agentic workflows become standard, expect the industry to converge on persistent KV cache support as a baseline requirement, with vendors likely competing on cache eviction policies and memory efficiency.
- Developers using Ollama for agent loops should consider implementing application-level prompt caching or switching to vLLM/SGLang for production workloads to avoid compounding latency and cost penalties.
Disclaimer: The above content is generated by AI and is for reference only.