BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
BHARATI introduces morphology-aware SentencePiece BPE tokenizers specifically designed for classical Indian languages (Sanskrit, Tamil, Telugu, Kannada, Malayalam) alongside English and Hindi. The v3 tokenizer achieves a subword fertility of 2.6 tokens per Indian Knowledge System technical term, significantly outperforming GPT-2's 5.25 and the mBART-50 baseline's 3.75. On a held-out IKS test set, BHARATI v3 reduces sequence length by approximately 90% compared to byte-level encodings and roughly
Analysis
TL;DR
- BHARATI introduces morphology-aware SentencePiece BPE tokenizers specifically designed for classical Indian languages (Sanskrit, Tamil, Telugu, Kannada, Malayalam) alongside English and Hindi.
- The v3 tokenizer achieves a subword fertility of 2.6 tokens per Indian Knowledge System technical term, significantly outperforming GPT-2's 5.25 and the mBART-50 baseline's 3.75.
- On a held-out IKS test set, BHARATI v3 reduces sequence length by approximately 90% compared to byte-level encodings and roughly 25% relative to the mBART-50 multilingual baseline.
- The system supports native script coverage across all seven target languages within a balanced 781 MB corpus, addressing inefficiencies caused by agglutinative morphology and productive sandhi in standard tokenizers.
Why It Matters
This work addresses a critical bottleneck in deploying Large Language Models for low-resource and morphologically complex Indian languages. By optimizing tokenization for specific linguistic features like sandhi and domain-specific vocabularies, BHARATI directly increases effective context length and computational efficiency, which is essential for building capable AI systems that preserve and process India's vast digital heritage and knowledge systems.
Technical Details
- Tokenizer Architecture: Utilizes SentencePiece Byte-Pair Encoding (BPE) trained on a curated, balanced corpus rather than relying on general-purpose web data.
- Iterative Development: Three versions were developed: v1 focused on English/Sanskrit with fallback for Tamil; v2 expanded to four languages; v3 achieved full native subword coverage for all seven languages (English, Hindi, Sanskrit, Tamil, Telugu, Kannada, Malayalam).
- Vocabulary Size: The final model employs a vocabulary size of 32,000 tokens.
- Evaluation Metrics: Performance measured via subword fertility (tokens per technical term) and sequence length reduction on a dataset of 490 IKS-domain sentences (70 per language).
- Linguistic Handling: Specifically engineered to handle agglutinative morphology and phonological fusion (sandhi) which typically cause standard tokenizers to fragment words excessively.
Industry Insight
- Efficiency Gains: Reducing sequence length by 90% relative to byte-level encoding allows for significantly longer context windows or reduced training costs for downstream models processing Indic texts.
- Domain Specialization: The success of reserving specific terms as single tokens suggests that hybrid approaches combining general BPE with domain-specific vocabulary injection are highly effective for specialized fields like Indian Knowledge Systems.
- Open Source Reproducibility: The release of training scripts and evaluation benchmarks provides a replicable framework for developing similar morphology-aware tokenizers for other under-resourced or morphologically complex language families globally.
Disclaimer: The above content is generated by AI and is for reference only.