OpenAI Releases GPT-Realtime-2.1 and GPT-Realtime-2.1-mini for Low-Latency Voice Agents in the API OpenAI发布GPT-Realtime-2.1和GPT-Realtime-2.1-mini，用于API中的低延迟语音代理

TL;DR

• OpenAI released GPT-Realtime-2.1 and GPT-Realtime-2.1-mini, focusing on low-latency voice and multimodal experiences via a single speech-to-speech model.
• The new mini model introduces configurable reasoning capabilities and tool use at the same price point as the previous generation, offering a cost-effective alternative for high-volume applications.
• P95 latency across Realtime voice models has been reduced by at least 25% through improved caching mechanisms, significantly enhancing user experience.
• Configurable reasoning effort levels allow developers to balance latency and computational cost, with "low" recommended for most production voice agents.
• Significant pricing advantages exist for cached inputs, particularly for audio, making long-session applications more economically viable.

Why It Matters

This release democratizes advanced agentic capabilities for real-time voice applications by integrating reasoning and tool use into the cost-efficient mini tier, allowing developers to build complex, multi-step voice assistants without prohibitive costs. The substantial reduction in p95 latency addresses a critical pain point in voice AI, ensuring smoother, more natural conversations that are essential for consumer adoption. Furthermore, the introduction of configurable reasoning effort provides a practical lever for optimizing the trade-off between intelligence and performance, enabling tailored solutions for diverse use cases ranging from simple queries to complex troubleshooting.

Technical Details

• Model Architecture: Utilizes a single end-to-end speech-to-speech model within the Realtime API, eliminating the need for separate speech-to-text and text-to-speech chains, which reduces latency and preserves vocal nuance.
• Reasoning and Tool Use: Both models support function calling and internal reasoning before speaking. The mini model allows for configurable reasoning effort (minimal, low, medium, high, xhigh), with "low" being the default to minimize latency.
• Latency Optimization: Achieved a minimum 25% reduction in p95 latency through enhanced caching strategies, specifically benefiting the tail-end response times that users perceive as lag.
• Pricing Structure: Costs are segmented by input/output type (text, audio, image) and cache status. For example, cached audio input for the mini model is priced at $0.30 per 1M tokens, compared to $10.00 for fresh input, while audio output remains at $20.00 per 1M tokens.
• Implementation: Supports direct browser connections via WebRTC with ephemeral client secrets, server-side media pipelines using WebSockets, and telephony integration via SIP.

Industry Insight

Developers should prioritize the GPT-Realtime-2.1-mini model for high-throughput customer support or in-app assistants where cost efficiency and speed are paramount, leveraging the cached input discounts to further reduce operational expenses. The ability to configure reasoning effort offers a strategic advantage in tuning performance; teams should start with "low" effort for standard interactions and only increase complexity for scenarios requiring deep analysis, thereby managing token usage and latency effectively. Additionally, the integration of spoken preambles during tool execution mitigates user confusion caused by silence, setting a new standard for conversational coherence in agentic voice interfaces.

TL;DR

• OpenAI发布GPT-Realtime-2.1及轻量版Mini，专为低延迟语音和多模态交互设计，支持端到端语音处理。
• Mini模型引入内部推理能力并支持工具调用，保持与旧版相同成本，同时通过优化缓存使P95延迟降低至少25%。
• 提供可配置的推理努力级别以平衡延迟与性能，缓存机制显著降低长会话成本，Mini版音频输出价格约为全模型的三分之一。
• 新增改进的字母数字识别、静音/噪音处理及中断行为，适用于客服、预约调度等需要高实时性的场景。

为什么值得看

此次更新解决了语音Agent在工具调用时的“静默停顿”痛点，通过内部推理和前导语音保持对话连贯性，极大提升了用户体验。对于开发者而言，新增的低成本推理模型和显著的延迟优化，使得构建大规模、高并发且具备复杂逻辑的实时语音应用在经济和技术上更加可行。

技术解析

• 架构与延迟优化：采用单模型端到端语音处理，避免传统STT/TTS链式调用的延迟。通过改进缓存机制，将P95尾部分位延迟降低至少25%，其中缓存音频输入成本大幅降至$0.30/1M tokens。
• 推理与工具集成：Mini模型支持内部推理（Reasoning），允许模型在回答前进行思考，并通过“前导语音”（如“我正在查询...”）在调用函数时保持用户连接感。推理努力程度可配置为minimal至xhigh五个等级。
• 模型规格与定价：GPT-Realtime-2.1增强了指令遵循和字母数字识别；GPT-Realtime-2.1-mini主打高性价比，音频输出价格为$20.00/1M tokens，仅为标准版的约1/3，但保留了核心推理和工具调用能力。
• 实现细节：浏览器端通过WebRTC直连API，服务端使用WebSocket或SIP。需生成短期客户端密钥以保护主API Key，会话配置中可指定模型、指令、推理层级及具体工具函数。

行业启示

• 实时交互体验升级：语音Agent不再仅是简单的问答机器人，具备“思考”和“预告”能力使其能处理更复杂的任务流，未来将成为智能客服和虚拟助手的标配能力。
• 成本结构重塑：缓存机制带来的成本骤降（尤其是音频输入）改变了实时语音应用的计费模型，使得高频、长时长的语音交互在商业上更具可持续性。
• 开发范式转变：从拼接多个专用模型转向单一多模态模型，简化了系统架构，降低了维护复杂度，开发者应优先评估单模型方案在延迟和一致性上的优势。