Comparative Analysis of Multilingual Pre-trained Models for Nepali Automatic Speech Recognition
First controlled benchmark comparing six multilingual pretrained ASR models on Nepali under identical fine-tuning protocols, using the OpenSLR SLR54 corpus (~165 hours) Whisper-Large-v3-Turbo (14.76% WER) and IndicWav2Vec (14.89% WER) achieve near-identical top performance despite a 9x parameter gap and 40x pretraining data gap CTC-based decoders run up to 29x faster than autoregressive Whisper models at equivalent accuracy, strongly favoring CTC for latency-sensitive deployments Massively multi
Analysis
TL;DR
- First controlled benchmark comparing six multilingual pretrained ASR models on Nepali under identical fine-tuning protocols, using the OpenSLR SLR54 corpus (~165 hours)
- Whisper-Large-v3-Turbo (14.76% WER) and IndicWav2Vec (14.89% WER) achieve near-identical top performance despite a 9x parameter gap and 40x pretraining data gap
- CTC-based decoders run up to 29x faster than autoregressive Whisper models at equivalent accuracy, strongly favoring CTC for latency-sensitive deployments
- Massively multilingual MMS-1B shows the smallest out-of-domain degradation on FLEURS (+12.55 pp), demonstrating that scale improves robustness over peak in-domain accuracy
- Language-family proximity in pretraining can effectively substitute for raw model scale when targeting in-domain Nepali speech recognition
Why It Matters
This benchmark fills a critical gap for low-resource and underrepresented language communities by providing the first standardized, multi-model, efficiency-aware reference for Nepali ASR. For AI practitioners building speech systems for Indic languages, it offers actionable guidance on trading off accuracy, latency, and out-of-domain robustness. The finding that family-proximity can replace scale has broader implications for resource-constrained multilingual ASR development.
Technical Details
- Six pretrained models evaluated: XLSR-53, IndicWav2Vec, MMS-1B, Whisper-Medium, Whisper-Large-v3-Turbo, and Conformer-Hi, spanning CTC self-supervised, autoregressive encoder-decoder, and hybrid Conformer-CTC architectures
- Fine-tuning conducted on OpenSLR SLR54 Nepali corpus (~165 hours) with identical preprocessing, data splits, optimizer, and family-matched learning-rate schedules across all models
- Evaluation metrics include Word Error Rate (WER), Character Error Rate (CER), and Real-Time Factor (RTF) across three independent test sets: OpenSLR, FLEURS, and Common Voice
- Key quantitative results: Whisper-Large-v3-Turbo and IndicWav2Vec both achieve ~14.8% WER; CTC decoders achieve up to 29x speedup over autoregressive Whisper at comparable accuracy; MMS-1B shows +12.55 pp out-of-domain degradation on FLEURS, the smallest among all models tested
Industry Insight
- For production Nepali ASR systems, CTC-based architectures (e.g., XLSR-53, IndicWav2Vec) should be the default choice when latency is a constraint, as they match top accuracy at a fraction of the compute cost of large autoregressive models
- When targeting Indic languages with limited in-domain data, prioritizing models pretrained on language-family-proximate data (e.g., IndicWav2Vec for Nepali) can be more effective than scaling up parameter count alone
- For applications requiring strong out-of-domain generalization across diverse speech conditions, massively multilingual models like MMS-1B offer superior robustness, making them suitable for deployment in noisy or variable real-world environments
Disclaimer: The above content is generated by AI and is for reference only.