Efficiently Adapting Spoken Language Models for the Singaporean Context
The study introduces HT-Moonstone (5B), a Spoken Language Model adapted for the Singaporean Home Team context, demonstrating that efficient adaptation can rival much larger models. A novel multi-task objective adapts the CoBa reweighting scheme to speech, combined with LoRA fine-tuning and a surrogate text-QA dataset to prevent catastrophic forgetting. The authors release HTD-multilingual-QA, a substantial 504,853-sample multilingual QA dataset in both text and spoken forms covering Singapore's
Analysis
TL;DR
- The study introduces HT-Moonstone (5B), a Spoken Language Model adapted for the Singaporean Home Team context, demonstrating that efficient adaptation can rival much larger models.
- A novel multi-task objective adapts the CoBa reweighting scheme to speech, combined with LoRA fine-tuning and a surrogate text-QA dataset to prevent catastrophic forgetting.
- The authors release HTD-multilingual-QA, a substantial 504,853-sample multilingual QA dataset in both text and spoken forms covering Singapore's four official languages.
- HT-Moonstone achieves state-of-the-art accent and gender recognition while maintaining over 98% of its original speech QA capabilities, outperforming models up to 7x its size.
Why It Matters
This research addresses a critical gap in adapting Spoken Language Models to sensitive, domain-specific contexts where original training data is inaccessible. By providing a replicable framework for multilingual spoken adaptation and releasing a large-scale dataset, it offers a practical pathway for government and enterprise applications requiring robust, low-resource speech reasoning.
Technical Details
- Model Architecture & Adaptation: Utilizes a 5B parameter open-source SLM adapted via Low-Rank Adaptation (LoRA) to handle five specific speech tasks across Singapore's four official languages.
- Training Strategy: Employs a multi-task objective that modifies the CoBa reweighting scheme for speech inputs and integrates a surrogate text-QA dataset to mitigate catastrophic forgetting during fine-tuning.
- Dataset Contribution: Introduces HTD-multilingual-QA, comprising 504,853 samples in both textual and spoken formats, facilitating robust multilingual spoken query interaction.
- Performance Metrics: The model matches or exceeds the performance of SLMs up to 7 times its size on most tasks, sets new benchmarks for accent and gender recognition, and incurs less than a 2% drop in general speech QA ability.
Industry Insight
- Organizations operating in multilingual, regulated environments should prioritize efficient fine-tuning techniques like LoRA combined with surrogate data strategies to maintain model stability without access to proprietary base training data.
- The release of HTD-multilingual-QA highlights the growing necessity for high-quality, parallel text-speech datasets in under-resourced linguistic contexts, encouraging investment in data collection for diverse dialects and accents.
- Developers can leverage adapted reweighting schemes like CoBa for speech to optimize training efficiency, potentially reducing computational costs while achieving competitive performance against significantly larger models.
Disclaimer: The above content is generated by AI and is for reference only.