NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling
NVIDIA released NemotronLabs VoiceChat 11B, an open 11B-parameter end-to-end speech-to-speech model that eliminates the traditional ASR→LLM→TTS cascade in favor of a unified full-duplex network The model achieves 448 ms smooth turn-taking latency and a 1.00 take-over rate at 480 ms, enabling natural barge-in conversation where users can interrupt mid-turn It is the first open full-duplex model to support tool calling via a separate side-channel output with operator-defined "on-hold" messages to
Analysis
TL;DR
- NVIDIA released NemotronLabs VoiceChat 11B, an open 11B-parameter end-to-end speech-to-speech model that eliminates the traditional ASR→LLM→TTS cascade in favor of a unified full-duplex network
- The model achieves 448 ms smooth turn-taking latency and a 1.00 take-over rate at 480 ms, enabling natural barge-in conversation where users can interrupt mid-turn
- It is the first open full-duplex model to support tool calling via a separate side-channel output with operator-defined "on-hold" messages to prevent conversational silence during API calls
- Despite permissive OpenMDW-1.1 licensing, NVIDIA labels the checkpoint "research purposes only" due to documented failure modes including a 2-minute audio context ceiling, multi-turn degradation into gibberish, and dropped transcription words
- The model requires a single GPU with at least 80 GB VRAM (A100/H100/RTX 6000 Pro/B200) and currently has no hosted API or inference provider
Why It Matters
This release represents a significant architectural shift in conversational AI, demonstrating that unified speech-to-speech models can compete with cascaded ASR-LLM-TTS pipelines on latency while enabling full-duplex interaction—a capability previously limited to closed proprietary systems. For AI practitioners, it provides the first open benchmark for evaluating full-duplex tool-calling voice agents, though the research-only designation and hardware requirements signal that production readiness remains a work in progress.
Technical Details
- Architecture: Hybrid Mamba/Transformer combining a Fast Conformer speech encoder (from Nemotron-Speech-Streaming-En-0.6b) for continuous 16 kHz audio encoding, the Nemotron Nano v2 LLM backbone for audio-to-text token prediction, an NVIDIA TTS decoder and codec for 22.05 kHz speech generation, and a separate output channel dedicated to
<TOOLCALL>scripts - Training: Approximately 550k hours of audio across real and synthetic corpora, built on SALM-Duplex and Audio Flamingo 3 datasets
- Tool-calling mechanism: Tool calls are emitted on a side channel as
<TOOLCALL>blocks; the system returns results via<TOOL_RESPONSE>blocks, with operator-defined on-hold messages spoken during API execution to maintain conversational flow - Performance benchmarks: Ranks #2 among open full-duplex models on VoiceBench and Full-Duplex-Bench 1.0; AU Harness BFCL-v3 spoken tool calling scores 56.1% average (58.5% simple, 62.5% multiple, 42.5% parallel); Full-Duplex-Bench v3 shows 82.5% tool selection, 44.2% argument accuracy, 33% pass@1
- Constraints: Maximum 5 tools per session, no reliable simultaneous multi-tool calls, no user interruption during tool execution, ASCII-only system prompts and tool responses required
Industry Insight
- The 448 ms turn-taking latency and full-duplex barge-in capability set a new open-source baseline for real-time voice agents, accelerating competitive pressure on proprietary systems like Google's Gemini Voice and Anthropic's voice features that previously dominated this space
- The explicit failure modes (context ceiling, multi-turn degradation, transcription drops) highlight that end-to-end speech models still struggle with long-context stability and error propagation—teams piloting this model should implement external monitoring and fallback to cascaded pipelines for production-critical applications
- The 80 GB GPU requirement and absence of a hosted API create a barrier to entry that favors well-funded organizations and research labs; expect inference optimization efforts (quantization, distillation) to emerge quickly, and anticipate third-party providers building managed serving layers within months
Disclaimer: The above content is generated by AI and is for reference only.