The Anatomy of an ASR Hallucination ASR幻觉的解剖

TL;DR

• ASR hallucinations are reframed as a symptom of broader grounding failure, where transcripts drift away from audio guidance
• The final encoder stage in Conformer-Large architectures acts as a critical boundary for grounded recognition, identified consistently across CTC and RNN-T decoder families
• Bypassing the final encoder block causes near-universal divergence under environmental degradation and speaker-background shift, while middle block bypassal has minimal impact
• At this terminal stage, representations become compact, grapheme information becomes explicit, and outputs become readable by the decoder
• The intervention produces garbled or repetitive output rather than fluent fabrication, indicating this stage is a mechanistic precondition for hallucination, not its complete origin

Why It Matters

This work provides the first mechanistic dissection of where and how ASR systems lose grounding to audio input, offering practitioners a concrete target for diagnosing and mitigating hallucinations in production speech recognition systems. By identifying the final encoder stage as a critical dependency across two independently trained architectures, it suggests that robustness improvements should focus on terminal-stage representation quality rather than earlier layers.

Technical Details

• Models studied: Two independently trained Conformer-Large recognizers — one using CTC decoding and one using RNN-T (Recurrent Neural Network Transducer) decoding — ensuring findings generalize across decoder families
• Intervention method: Selective bypassing of encoder blocks (final vs. middle stages) to isolate where grounding failure becomes possible
• Stress conditions: Environmental degradation and speaker-background shift applied to test robustness under distributional changes
• Key observation at final stage: Representations become more compact, text becomes readable by the trained decoder, and grapheme information becomes explicit — all hallmarks of the transition from acoustic to linguistic representation
• Output characterization: Bypassing the final block produces garbled or repetitive output rather than fluent fabricated text, distinguishing this as a grounding precondition rather than the full hallucination mechanism

Industry Insight

• Focus interpretability and robustness efforts on the terminal encoder stage of ASR pipelines, as this is where the acoustic-to-linguistic grounding boundary is most fragile under distribution shift
• When diagnosing hallucinations in production ASR systems, inspect whether the model's final encoder representations are maintaining adequate audio-linguistic alignment rather than assuming the issue originates in the decoder
• The finding that middle blocks are relatively dispensable suggests potential for efficient model compression or early-exit strategies in ASR systems without significant grounding loss, provided the final stage is preserved

TL;DR

• ASR幻觉本质上是音频“接地失败”（grounding failure）的表现，即转录文本逐渐脱离音频内容的实际引导。
• 研究对比了CTC与RNN-T两种解码器的Conformer-Large模型，发现编码器最终阶段是维持音频-文本对齐的关键边界。
• 绕过最终编码器块会导致几乎所有语音片段的输出发散，而绕过中间块影响甚微。
• 该阶段表征更紧凑、图素信息显式化，且模型干预仅产生混乱/重复输出而非流畅幻觉文本，说明其是幻觉的必要前提而非完整成因。
• 研究在环境噪声与说话人背景偏移双重分布偏移下，验证了终端阶段依赖的跨架构一致性。

为什么值得看

本文为ASR系统的可靠性与可解释性研究提供了清晰的机制视角，帮助从业者从“接地失败”而非单纯“模型错误”的角度理解幻觉问题。对工业界部署高鲁棒语音交互系统具有重要参考价值，明确了模型架构中需重点监控与加固的关键阶段。