Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu
The paper addresses hate speech detection in Roman Urdu, a low-resource language characterized by informal structure, high variation, and lack of standardized spelling Zero-shot LLM inference on Roman Urdu achieves only moderate performance (F1 = 0.56), highlighting the limitations of off-the-shelf models for low-resource languages Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) dramatically improves classification performance to F1 > 0.93 by updating only a small fractio
Analysis
TL;DR
- The paper addresses hate speech detection in Roman Urdu, a low-resource language characterized by informal structure, high variation, and lack of standardized spelling
- Zero-shot LLM inference on Roman Urdu achieves only moderate performance (F1 = 0.56), highlighting the limitations of off-the-shelf models for low-resource languages
- Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) dramatically improves classification performance to F1 > 0.93 by updating only a small fraction of trainable parameters
- The study benchmarks multiple transformer architectures including Mistral, LLaMA, Falcon, and multilingual BERT on the PURUTT dataset containing over 72,000 annotated comments
- PEFT demonstrates both outstanding accuracy and excellent computational efficiency, making it a highly viable strategy for low-resource language processing tasks
Why It Matters
This research directly addresses a critical gap in AI safety and content moderation for underrepresented languages, where the majority of existing hate speech detection systems are built exclusively for high-resource languages like English. For AI practitioners working on multilingual applications, the findings provide a practical blueprint for achieving state-of-the-art performance on low-resource languages without the prohibitive computational costs of full fine-tuning, enabling more inclusive and globally scalable content moderation systems.
Technical Details
- Dataset: PURUTT (Parallel Urdu and Roman Urdu Corpus for Toxic Comments and Transliteration) with over 72,000 annotated comments, specifically designed for hate speech detection in Roman Urdu
- Models Evaluated: Four transformer-based architectures — Mistral, LLaMA, Falcon, and multilingual BERT — compared under both zero-shot inference and PEFT (LoRA) fine-tuning regimes
- PEFT Approach: Low-Rank Adaptation (LoRA) applied to freeze the base model weights while training only low-rank decomposition matrices, significantly reducing the number of trainable parameters
- Performance Metrics: Zero-shot models achieved F1 = 0.56, while LoRA-finetuned models surpassed F1 > 0.93, representing a substantial performance gain with minimal parameter updates
- Key Challenge Addressed: Roman Urdu's high orthographic variation, lack of contextually consistent spellings, and informal language structure typical of South Asian social media communication
Industry Insight
- Organizations deploying content moderation systems in South Asian markets should prioritize PEFT-based adaptation over zero-shot approaches, as the performance gap (F1 0.56 vs. 0.93) is too significant to ignore for production-grade applications
- The PURUTT dataset and LoRA-based methodology provide a replicable template for extending hate speech detection to other low-resource languages, particularly those with informal digital writing conventions across the Global South
- As regulatory pressure mounts for multilingual AI safety compliance, investing in parameter-efficient adaptation pipelines for low-resource languages represents a high-ROI strategy that balances performance, cost, and inclusivity simultaneously
Disclaimer: The above content is generated by AI and is for reference only.