Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition
Dual-Form ASR (DF-ASR) is a novel framework that unifies spoken-form ASR and semantics-aware written-form inverse text normalization (ITN) into a single model, eliminating the error-prone cascaded pipeline. The framework uses paired spoken-form and written-form supervision, with training data constructed via an LLM-driven generate-and-judge workflow. ITN-MWER, a sequence-level training objective, is introduced to assign higher penalty costs to errors on normalization-sensitive spans. A decision-
Analysis
TL;DR
- Dual-Form ASR (DF-ASR) is a novel framework that unifies spoken-form ASR and semantics-aware written-form inverse text normalization (ITN) into a single model, eliminating the error-prone cascaded pipeline.
- The framework uses paired spoken-form and written-form supervision, with training data constructed via an LLM-driven generate-and-judge workflow.
- ITN-MWER, a sequence-level training objective, is introduced to assign higher penalty costs to errors on normalization-sensitive spans.
- A decision-aware REQUIRE-ITN/FORBID-ITN evaluation protocol is proposed to separately measure required normalization accuracy and forbidden-span preservation.
- On manually annotated Chinese subsets from SpeechIO, DF-ASR outperforms open-source ASR-ITN systems and remains competitive with strong closed-source baselines while preserving prompt-level control between output forms.
Why It Matters
This work addresses a fundamental limitation in modern ASR systems: the decoupling of acoustic modeling from text normalization, which has long been a bottleneck for producing accurate written-form transcripts—especially for semantically dependent numeric expressions in Chinese. By unifying spoken and written form generation within a single framework, DF-ASR offers a more robust and flexible architecture that practitioners can adopt to improve end-to-end speech-to-text pipelines without relying on fragile cascaded post-processing.
Technical Details
- Dual-form supervision: The model is trained with paired spoken-form and written-form targets, enabling it to learn normalization patterns directly from acoustic features rather than relying on a separate rewrite module.
- LLM-driven data construction: A generate-and-judge workflow powered by a large language model is used to construct high-quality dual-form training data, ensuring reliable supervision signals for ITN tasks.
- ITN-MWER objective: A sequence-level minimum word error rate variant that imposes higher cost on errors occurring within normalization-sensitive spans, directly optimizing for ITN accuracy during training.
- REQUIRE-ITN/FORBID-ITN protocol: A novel evaluation metric that separately quantifies (1) the model's ability to perform required normalizations and (2) its ability to preserve spans that should not be normalized, providing finer-grained diagnostic insight.
- Prompt-level selection: Despite unified training, the framework retains the ability to selectively output either spoken-form or written-form transcripts via prompt conditioning, preserving flexibility for downstream applications.
Industry Insight
- The cascaded ASR-ITN architecture remains the de facto standard in many production systems; DF-ASR's unified approach could drive a paradigm shift toward end-to-end dual-form models, reducing error propagation and simplifying system design.
- The LLM-driven generate-and-judge workflow for training data construction offers a reusable recipe for other language-specific ITN challenges, particularly for languages with complex numeric and temporal expressions like Chinese.
- The REQUIRE-ITN/FORBID-ITN evaluation protocol provides a more nuanced benchmarking framework that the community could adopt to better assess ITN quality beyond aggregate WER, encouraging more rigorous development of normalization-aware ASR systems.
Disclaimer: The above content is generated by AI and is for reference only.