NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages
NE-BERT is a multilingual encoder model trained on ~8.3 million sentences across 9 Northeast Indian languages plus Hindi and English as anchor languages The model uses weighted data sampling and a custom SentencePiece Unigram tokenizer to address vocabulary fragmentation in extremely low-resource languages NE-BERT achieves 15.97X and 7.64X lower average perplexity compared to IndicBERT-V2 and MuRIL respectively across all 9 languages Tokenization fertility is 1.50X better than mBERT, and downstr
Analysis
TL;DR
- NE-BERT is a multilingual encoder model trained on ~8.3 million sentences across 9 Northeast Indian languages plus Hindi and English as anchor languages
- The model uses weighted data sampling and a custom SentencePiece Unigram tokenizer to address vocabulary fragmentation in extremely low-resource languages
- NE-BERT achieves 15.97X and 7.64X lower average perplexity compared to IndicBERT-V2 and MuRIL respectively across all 9 languages
- Tokenization fertility is 1.50X better than mBERT, and downstream POS tagging evaluation validates practical utility on three languages
- All resources (model, test sets, training corpus) are released under CC-BY-4.0 for open research and digital inclusion
Why It Matters
This work addresses a critical gap in multilingual NLP by focusing on Northeast Indian languages that are severely underrepresented in existing models, demonstrating that targeted training on low-resource languages can dramatically outperform generalist multilingual models. For AI practitioners working on underrepresented languages, it provides a replicable methodology combining aggressive upsampling, custom tokenization, and domain-specific pretraining. The open release of all resources supports broader efforts in digital inclusion and linguistic diversity in AI.
Technical Details
- Architecture & Training: NE-BERT is a BERT-style multilingual encoder trained on approximately 8.3 million sentences spanning 9 Northeast Indian languages (including extremely low-resource languages like Pnar with only 1,002 sentences and Kokborok with 2,463 sentences) plus 2 anchor languages (Hindi and English)
- Tokenization: A custom SentencePiece Unigram tokenizer was developed to reduce vocabulary fragmentation, achieving 1.50X better tokenization fertility than multilingual BERT (mBERT)
- Data Sampling Strategy: Weighted data sampling with aggressive upsampling was employed to address the extreme data scarcity in low-resource languages, ensuring they receive sufficient training signal despite minimal corpus sizes
- Evaluation: Perplexity benchmarks show NE-BERT outperforming IndicBERT-V2 (15.97X lower) and MuRIL (7.64X lower) across all 9 languages; downstream POS tagging evaluation was conducted on three Northeast Indian languages to validate practical utility
- Open Release: Model weights, test sets, and training corpus released under CC-BY-4.0 license
Industry Insight
- The success of NE-BERT demonstrates that domain-specific, low-resource multilingual models can significantly outperform generalist multilingual models, suggesting a strategic shift toward region- and language-specific pretraining for underrepresented communities rather than relying solely on large-scale multilingual models
- Aggressive upsampling and custom tokenization emerge as practical, cost-effective techniques for addressing vocabulary fragmentation in extremely low-resource settings, offering a blueprint for similar efforts globally
- The open release of both model and training data under CC-BY-4.0 sets a precedent for community-driven NLP development, encouraging collaborative efforts to digitize and preserve linguistically diverse languages through AI
Disclaimer: The above content is generated by AI and is for reference only.