Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions
Proposes Hybrid Search, a targeted self-correction strategy for LLM-based ASR that leverages preserved base LLMs during LoRA-adapted inference Introduces interaction features measuring the relationship between ASR hidden states and base-LLM hidden states as signals for token-level semantic dependence Selectively refining high-dependence tokens significantly outperforms naive global correction methods like rescoring and late fusion Demonstrates that warm-initialized LLM-based ASR models retain la
Analysis
TL;DR
- Proposes Hybrid Search, a targeted self-correction strategy for LLM-based ASR that leverages preserved base LLMs during LoRA-adapted inference
- Introduces interaction features measuring the relationship between ASR hidden states and base-LLM hidden states as signals for token-level semantic dependence
- Selectively refining high-dependence tokens significantly outperforms naive global correction methods like rescoring and late fusion
- Demonstrates that warm-initialized LLM-based ASR models retain latent capacity to benefit from their pre-adaptation base LLMs at inference time
- Bridges the gap between external logit fusion and internal warm initialization strategies in LLM-integrated ASR systems
Why It Matters
This work addresses a critical practical question for AI practitioners building speech recognition systems: how to maximize the utility of large language models without discarding the semantic knowledge embedded in the base LLM during adaptation. The Hybrid Search approach offers a computationally efficient alternative to expensive rescoring or late fusion pipelines, making it more viable for production ASR systems. For researchers, it opens a new direction in understanding how hidden-state interactions can serve as reliable signals for targeted model refinement.
Technical Details
- Hybrid Search mechanism: A targeted correction strategy that identifies tokens with high semantic dependence by computing interaction features between the LoRA-adapted ASR model's hidden states and the preserved base LLM's hidden states, then selectively refines only those tokens rather than applying global correction.
- LoRA-adapted setting: The approach operates under low-rank adaptation where the base LLM weights are preserved alongside the adapted ASR weights, enabling the model to access both adapted and pre-adaptation representations simultaneously during inference.
- Interaction features: Novel features characterizing the relationship/distance between ASR hidden states and base-LLM hidden states serve as proxies for semantic dependence, guiding which tokens benefit most from base-LLM correction.
- Benchmark comparisons: Evaluated against global LLM-correction baselines including rescoring and late fusion, demonstrating that selective token refinement yields substantially better ASR performance than naive full-sequence correction approaches.
Industry Insight
- Organizations deploying LLM-based ASR should consider preserving base model weights during adaptation rather than fully fine-tuning, as the retained base LLM can provide meaningful inference-time corrections through interaction-based refinement.
- Targeted, token-level correction strategies like Hybrid Search offer a compelling efficiency-accuracy tradeoff for production speech systems, reducing computational overhead compared to full rescoring pipelines while achieving superior results.
- The finding that hidden-state interactions encode useful semantic dependence signals suggests broader applicability beyond ASR—similar approaches could benefit other modalities where LLMs are adapted for specialized tasks while retaining base model knowledge.
Disclaimer: The above content is generated by AI and is for reference only.