Lowest-Latency Inference APIs for Voice and Realtime Agents: A Time to First Token TTFT-First Benchmark
TTFT (Time to First Token) is an insufficient metric for voice agents because TTS models require complete sentences before generating audio, making TTFS (Time to First Sentence) the more relevant user-experience metric The practical end-to-end latency budget for natural voice conversation is 700ms–1.2s, with a target median of 800ms voice-to-voice latency and roughly 700ms of TTFT budget for the LLM layer alone Silicon-optimized providers (Cerebras, Groq) excel at throughput but may have higher
Analysis
TL;DR
- TTFT (Time to First Token) is an insufficient metric for voice agents because TTS models require complete sentences before generating audio, making TTFS (Time to First Sentence) the more relevant user-experience metric
- The practical end-to-end latency budget for natural voice conversation is 700ms–1.2s, with a target median of 800ms voice-to-voice latency and roughly 700ms of TTFT budget for the LLM layer alone
- Silicon-optimized providers (Cerebras, Groq) excel at throughput but may have higher TTFT, while diffusion-based models like Mercury 2 achieve high token speeds (770 tok/s) but suffer catastrophic TTFT (3.07s), making them unsuitable for voice
- Benchmark methodology significantly impacts results: workload shape (10k vs 1k tokens), server location, reasoning token definitions, and measurement boundaries all affect reported TTFT numbers
- Frontier models (GPT-5.6 Luna, Gemini 3.7 Flash, Claude 4.5 Haiku) generally fall in the 0.59s–0.90s TTFT range, with output speeds varying dramatically from 82 to 315 tokens per second
Why It Matters
Voice AI is rapidly moving from novelty to production deployment, and latency is the single most critical factor determining whether an agent feels conversational or frustratingly sluggish. AI practitioners building voice agents need to understand that optimizing for TTFT alone can produce agents that feel slow to users, since the complete pipeline from speech input to speech output involves multiple sequential and parallel stages that each consume part of a tight latency budget.
Technical Details
- Latency breakdown: A voice turn consists of STT (100–200ms), LLM inference with streaming (300–500ms), TTS (100–200ms), and network/WebRTC transport (50–150ms), totaling a practical target of 700ms–1.2s end-to-end
- TTFT vs TTFS distinction: TTFT measures when generation starts; TTFS (coined by LiveKit) measures when the first complete sentence of audio is produced, accounting for the TTS model's need for full clauses before synthesis
- Benchmark methodology: Artificial Analysis uses 10k input token prompts (changed from 1k in March 2026), tests from Google Cloud us-central1-a zone, and measures from the receiving side (request send to first usable token off API), with TTFT for reasoning models capturing the first reasoning token separately from the first answer token
- Provider performance tiers: Baseten's gpt-oss-120b leads with 0.23s TTFT and 266 tok/s output; Cerebras achieves exceptional throughput (1,697 tok/s) with 0.49s TTFT; Mercury 2's diffusion approach yields 3.07s TTFT despite 770 tok/s speed, making it four times over the LLM budget
- Human baseline: Typical human conversational response time is ~500ms, with pauses beyond 800ms feeling unnatural, establishing the psychological threshold for acceptable agent latency
Industry Insight
- Voice agent developers should prioritize providers that balance both low TTFT and high throughput rather than optimizing for either metric in isolation, as the combination determines TTFS which directly impacts perceived responsiveness
- The shift toward longer context prompts (10k tokens) in benchmarks reflects real production workloads where policy, persona, tool schemas, and retrieved data are front-loaded, meaning providers optimized for short prompts may underperform in actual deployments
- Silicon vendors building for raw throughput (Cerebras, Groq) are well-positioned for voice use cases where fast sentence completion matters, while novel architectures like diffusion-based LMs (Mercury 2) need to address TTFT before becoming viable for real-time voice applications
Disclaimer: The above content is generated by AI and is for reference only.