NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone
NVIDIA released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text LLM that processes and generates both speech and general audio while preserving strong text intelligence. The model utilizes a 30B-parameter Mixture-of-Experts (MoE) architecture with 3B active parameters, employing a hybrid Mamba-Transformer backbone and a simple unified design where audio tokens are treated uniformly with text tokens. Audex avoids the typical "text tax" regression seen in multimodal models through a mult
Analysis
TL;DR
- NVIDIA released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text LLM that processes and generates both speech and general audio while preserving strong text intelligence.
- The model utilizes a 30B-parameter Mixture-of-Experts (MoE) architecture with 3B active parameters, employing a hybrid Mamba-Transformer backbone and a simple unified design where audio tokens are treated uniformly with text tokens.
- Audex avoids the typical "text tax" regression seen in multimodal models through a multi-stage Supervised Fine-Tuning (SFT) curriculum followed by text-only Cascade Reinforcement Learning (RL) and Multi-Domain On-Policy Distillation (MOPD).
- It achieves state-of-the-art performance among open models on several benchmarks, including leading on OpenASR (6.82 WER), MMAU, and Audio Entailment, while maintaining competitive text reasoning scores comparable to its text-only backbone.
- The model supports a context length of 1 million tokens, runs on standard stacks like Megatron-LM and vLLM, and is available under a non-commercial license, enabling diverse applications from multilingual call centers to accessibility tools and sound design.
Why It Matters
Audex addresses a critical bottleneck in multimodal AI: the degradation of core language capabilities when audio or vision modalities are added. By demonstrating that a unified model can maintain high-fidelity text reasoning while adding robust audio understanding and generation, it offers a more efficient and scalable alternative to stacked, specialized models. This approach simplifies deployment for practitioners seeking end-to-end audio-text solutions without sacrificing the intelligence required for complex reasoning tasks.
Technical Details
- Architecture: Audex is built on the Nemotron-Cascade-2-30B-A3B backbone, a hybrid Mamba-Transformer with 52 layers, 128 routable experts, and 6 activated experts per token. It uses a single decoder stack rather than separate "thinker" and "talker" modules.
- Audio Integration: Audio inputs are encoded using AF-Whisper (based on Whisper Large-v3) and projected into the text embedding space via two-layer MLP adapters. Outputs use discrete audio tokens from X-Codec2 (speech, 50 tps) and X-Codec (non-speech, 200 tps), expanding the vocabulary from ~131k to ~205k tokens.
- Training Strategy: The model employs a multi-stage SFT curriculum (text SFT -> audio warmup with frozen text embeddings -> audio generation -> audio understanding) to prevent catastrophic forgetting. Post-SFT optimization uses text-only Cascade RL and MOPD, which improved text scores while maintaining audio performance.
- Performance Metrics: On text benchmarks, Audex scores 86.4 on MMLU-Redux and 92.2 on HMMT Feb25. For audio, it achieves a 6.82 Word Error Rate (WER) on OpenASR, 75.6 on MMAU, and 95.0 on Audio Entailment, outperforming competitors like Qwen3-Omni and Step-Audio-R1.1 in specific categories.
- Deployment: Supports 1M token context length, operates in instruct and thinking modes, and is compatible with vLLM 0.20.0 for inference, facilitating easy integration into existing LLM infrastructure.
Industry Insight
- Unified Architectures Over Stacked Models: The success of Audex suggests that future multimodal systems may increasingly favor unified architectures over complex cascades of specialized models, reducing latency and operational complexity.
- Preservation of Core Intelligence: The technique of using text-only RL after multimodal SFT provides a viable blueprint for integrating new modalities without degrading foundational language capabilities, a key concern for enterprise AI adoption.
- General Audio Generation as a Differentiator: Unlike many competitors focused solely on speech, Audex's ability to generate general audio (e.g., sound effects) opens new avenues for creative industries, gaming, and immersive media, creating a broader market opportunity for open-source audio models.
Disclaimer: The above content is generated by AI and is for reference only.