Voice Memory for Agentic Speech Recognition 语音记忆用于代理语音识别

TL;DR

• Voice Memory is an inference-only agentic speech recognition scheme using a frozen corrector and asynchronous score-gated optimizer.
• It adopts a split listener-thinker architecture coupled only through memory, enabling auditable and portable learned skills without weight changes.
• Restraint is the key discovered skill: unconstrained generative error correction over-corrects (breaking 64% of edits on financial news), while Voice Memory reduces this to 35%.
• Across ten HyPoradise domains, Voice Memory lowers weighted word error rate from 8.36% to 7.52% (7.47% with three in-context examples) without regressing any dataset below its 1-best baseline.
• Significant gains are observed in air-travel commands (8.40% → 3.40%) and noisy far-field speech (CHiME-4: 12.69% → 10.46%).
• The memory transfers across corrector families and adds zero parameters to the inference path.

Why It Matters

This work introduces a novel, lightweight framework for improving agentic speech recognition without retraining or modifying model weights, making it highly deployable in real-time systems where stability and auditability are critical. By decoding correction into a separate, constrained process, it avoids the pitfalls of aggressive generative correction while achieving measurable WER reductions across diverse domains—offering a practical path forward for robust, production-grade ASR systems.

Technical Details

• Architecture: Split listener-thinker design where a frozen corrector (listener) evaluates hypotheses per utterance, and a score-gated optimizer (thinker) asynchronously revises them via bounded edits only if they improve held-out scores.
• Mechanism: Uses a single per-domain memory file to store correction rules; decisions are made at stream time without changing model weights, ensuring portability and auditability.
• Error Correction Strategy: Unlike unconstrained generative error correction (GER), which breaks correct tokens in up to 64% of edits on financial data, Voice Memory applies restraint, reducing erroneous corrections to 35%.
• Evaluation: Tested across ten HyPoradise domains; achieves consistent WER reduction without degrading performance on any domain relative to its 1-best baseline.
• Transferability: The memory component generalizes across different corrector families and incurs no additional parameter overhead during inference.

Industry Insight

Agentic speech recognition systems can benefit significantly from adopting restrained, memory-based correction pipelines that avoid over-correction risks inherent in fully generative approaches. This approach enables plug-and-play improvement of existing ASR models without retraining or architectural changes, making it ideal for regulated or high-stakes environments like finance, healthcare, or aviation where reliability and traceability are paramount. Future implementations should consider integrating in-context examples as demonstrated, which further refine performance with minimal computational cost.

TL;DR

• 提出 Voice Memory，一种仅用于推理的代理语音识别方案，通过冻结校正器和异步优化器实现分听者-思考者架构。
• 引入“克制”作为核心技能，显著降低生成式错误修正（GER）的过度修正率（从64%降至35%），同时保持或提升词错误率（WER）。
• 在多个领域（如航空旅行命令、嘈杂远场语音）上实现显著性能提升，且无需增加推理参数，具备可审计性和可移植性。

为什么值得看

该研究为语音识别中的错误修正提供了一种高效、轻量级的推理优化路径，特别适合对延迟敏感或资源受限的代理系统。其“分听者-思考者”架构与克制机制的设计思路，也为其他模态的自修正系统提供了重要参考。

技术解析

• 采用“分听者-思考者”架构：听者（ASR模型）生成初始假设，思考者（冻结校正器）决定是否采纳或保留1-best结果；两者仅通过内存交互，无权重更新，确保模型可审计与可迁移。
• 使用分数门控优化器进行异步编辑：仅在严格提升验证集得分时才接受编辑，避免盲目修正导致正确内容被破坏。
• 基于经典ASR-LM框架扩展，支持域内单URL上下文输入，适用于流式场景下的实时决策。
• 实验覆盖HyPoradise十个领域，加权WER从8.36%降至7.52%，在特定任务如航空指令（8.40%→3.40%）和CHiME-4噪声语音（12.69%→10.46%）中表现突出。
• 不增加任何推理参数，且校正器可跨家族复用，具备良好的工程落地潜力。

行业启示

• 在语音代理系统中引入“选择性修正”机制比全量生成式纠错更有效，应优先探索基于规则或评分约束的轻量级后处理策略。
• “克制”作为一种隐式能力，可通过简单反馈循环自动习得，未来可推广至文本、图像等多模态任务的自修正流程。
• 架构解耦设计（听者+思考者+内存）有利于模块化部署与维护，适合构建高可靠、易迭代的AI语音服务系统。