Interfaze Ships diffusion-gemma-asr-small, an Open-Source Diffusion ASR Model Transcribing Six Languages via DiffusionGemma’s Parallel Denoising Decoder
Interfaze open-sources diffusion-gemma-asr-small, the first multilingual audio diffusion ASR model using a frozen 26B DiffusionGemma backbone and a lightweight 42M parameter adapter. The model utilizes uniform, random-token diffusion rather than absorbing mask schemes, enabling parallel transcription across six languages (English, German, French, Spanish, Hindi, Mandarin) via a single adapter. It achieves a 6.6% Word Error Rate (WER) on LibriSpeech test-clean, outperforming Whisfusion (8.3%) but
Analysis
TL;DR
- Interfaze open-sources diffusion-gemma-asr-small, the first multilingual audio diffusion ASR model using a frozen 26B DiffusionGemma backbone and a lightweight 42M parameter adapter.
- The model utilizes uniform, random-token diffusion rather than absorbing mask schemes, enabling parallel transcription across six languages (English, German, French, Spanish, Hindi, Mandarin) via a single adapter.
- It achieves a 6.6% Word Error Rate (WER) on LibriSpeech test-clean, outperforming Whisfusion (8.3%) but trailing autoregressive Whisper-small (~3.4%), with performance scaling determined by denoising steps rather than transcript length.
- Training required overcoming initial gradient stagnation by supervising the audio projector with CTC loss, allowing the frozen backbone to effectively attend to audio features extracted by a frozen Whisper-small encoder.
Why It Matters
This development challenges the dominance of autoregressive models in speech recognition by demonstrating that diffusion-based parallel decoding can achieve competitive accuracy while offering distinct advantages in batch processing and multilingual handling. For researchers, it provides a reproducible baseline for integrating audio modalities into large language models without full fine-tuning, highlighting the potential of hybrid architectures that combine specialized encoders with generative diffusion decoders.
Technical Details
- Architecture: Combines a frozen Whisper-small encoder (feature extractor) with a frozen DiffusionGemma-26B MoE backbone (activates 4B parameters). A trainable ~42M parameter adapter includes a projector that compresses 1500 acoustic frames into 188 audio tokens, which are scattered into the LLM's prompt slots.
- Diffusion Mechanism: Uses discrete uniform random-token diffusion where a fixed-length canvas is filled with random vocabulary tokens. Each denoising step locks confident predictions and re-randomizes the rest, running bidirectionally over approximately 16 steps.
- Training Strategy: Initial training failed due to gradient vanishing; the solution involved applying Connectionist Temporal Classification (CTC) loss directly to the audio tokens projected through the frozen language model head, stabilizing learning and reducing WER significantly.
- Performance Metrics: On LibriSpeech test-clean, it scores 6.6% WER. Inference speed is highly dependent on denoising steps: 8 steps yield 14.9x real-time speed with 15.7% WER on FLEURS-en, while 16 steps offer 10.3x speed with 15.6% WER.
Industry Insight
The decoupling of transcription cost from transcript length suggests significant efficiency gains for batch processing pipelines, where long audio clips do not incur proportionally higher computational costs compared to autoregressive methods. Additionally, the ability to handle six languages with a single small adapter reduces infrastructure complexity and model management overhead for multilingual applications, though practitioners must weigh the slight accuracy deficit against autoregressive baselines like Whisper.
Disclaimer: The above content is generated by AI and is for reference only.