SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
SpeakPay introduces a voice-first digital wallet for Nepal, addressing inaccessibility of mobile payment apps for visually impaired users The authors release NepFinSpeech-403, a 403-utterance dataset of Nepali financial voice commands covering 237 unique numerals across send, load, and balance operations Fine-tuning Whisper large-v2 with LoRA reduces Word Error Rate from 129.95% to 42.58% (67.2% relative reduction) and improves Devanagari numeral recognition from 0.0% to 73.9% Task-level impact
Analysis
TL;DR
- SpeakPay introduces a voice-first digital wallet for Nepal, addressing inaccessibility of mobile payment apps for visually impaired users
- The authors release NepFinSpeech-403, a 403-utterance dataset of Nepali financial voice commands covering 237 unique numerals across send, load, and balance operations
- Fine-tuning Whisper large-v2 with LoRA reduces Word Error Rate from 129.95% to 42.58% (67.2% relative reduction) and improves Devanagari numeral recognition from 0.0% to 73.9%
- Task-level impact far exceeds word-level metrics: Transaction Success Rate improves from 1.67% to 33.33%, a roughly 20x gain
- As few as 100 domain-specific utterances halve the zero-shot WER, with performance plateauing around 300 examples
Why It Matters
This work demonstrates that domain-adaptive fine-tuning of large pre-trained speech models can dramatically improve performance on low-resource languages and specialized domains, making it directly relevant for practitioners working on accessibility and financial technology in underrepresented linguistic contexts. The finding that word-level metrics substantially understate practical impact is a critical reminder for the community to evaluate speech recognition systems using task-oriented metrics rather than relying solely on WER.
Technical Details
- Model: Whisper large-v2 fine-tuned with Low-Rank Adaptation (LoRA) for domain adaptation to Nepali financial speech
- Dataset: NepFinSpeech-403, comprising 403 utterances of Nepali financial voice commands (send, load, balance operations) spanning 237 unique numerals in Devanagari script
- Performance: Zero-shot WER of 129.95% reduced to 42.58% after fine-tuning; Devanagari numeral recognition accuracy improved from 0.0% to 73.9%; Transaction Success Rate increased from 1.67% to 33.33%
- Data efficiency: Performance halves with as few as 100 utterances and plateaus around 300 examples, indicating strong data efficiency of LoRA fine-tuning
- Error analysis: Systematic numeral confusion patterns identified, including zero insertion/deletion and prefix hallucination, which account for the majority of remaining transaction failures; statistical significance confirmed via sign test (p < 10^-17)
Industry Insight
- Low-resource language speech systems can achieve transformative practical improvements through lightweight domain adaptation (LoRA) rather than requiring massive labeled datasets, lowering the barrier for inclusive AI deployment in emerging markets
- Researchers and practitioners should adopt task-level evaluation metrics (e.g., Transaction Success Rate) alongside traditional WER, as word-level errors can disproportionately impact real-world application performance, especially in numeracy-critical domains like finance
- The data efficiency findings suggest that organizations can achieve meaningful speech recognition improvements with minimal domain-specific data, enabling rapid prototyping and deployment of voice-first interfaces for accessibility-focused products
Disclaimer: The above content is generated by AI and is for reference only.