Cross Lingual Transfer in Tulu Legal Comprehension: Script-Dependent Improvement and RAG-Induced Knowledge Conflict
Cross-lingual transfer using transliteration across Dravidian scripts enables preliminary legal comprehension for low-resource languages like Tulu without large-scale training Kannada script transliteration produced the strongest positive trend among tested Dravidian scripts, indicating script-dependent comprehension variation RAG retrieval from Kannada legal papers yielded mixed results, with failures primarily manifesting as fact substitution and confabulation rather than corpus deficiencies M
Analysis
TL;DR
- Cross-lingual transfer using transliteration across Dravidian scripts enables preliminary legal comprehension for low-resource languages like Tulu without large-scale training
- Kannada script transliteration produced the strongest positive trend among tested Dravidian scripts, indicating script-dependent comprehension variation
- RAG retrieval from Kannada legal papers yielded mixed results, with failures primarily manifesting as fact substitution and confabulation rather than corpus deficiencies
- Model parsing and reasoning, not corpus content, were identified as the root cause of reasoning failures in low-resource multilingual settings
- A statistical-honesty framework and reasoning-trace analysis were deployed as broadly applicable evaluation techniques for low-resource multilingual RAG systems
Why It Matters
This research addresses a critical gap in multilingual AI: how to reliably evaluate and improve cross-lingual transfer for low-resource languages in specialized domains like law. For practitioners building multilingual RAG systems, the findings reveal that script choice and model reasoning robustness are tightly coupled, and that failure modes in low-resource settings stem from internal model processing rather than retrieval quality—insights essential for deploying AI in legally and linguistically diverse contexts.
Technical Details
- Three models (Llama3, Hex-1, Sarvam) were evaluated on classifying legal complaints written in Tulu, a low-resource Dravidian language, using transliteration across Dravidian scripts as a cross-lingual scaffold
- The study tested script-dependent comprehension by transliterating Tulu queries into other Dravidian scripts, with Kannada producing the strongest positive performance trend
- A RAG framework was implemented using a corpus of Kannada legal papers, revealing that retrieval alone did not guarantee improved comprehension and could introduce knowledge conflicts
- Two primary failure axes were identified: fact substitution (models fixating on specific passage excerpts that skewed reasoning) and confabulation (hallucinations with no basis in query or corpus)
- Reasoning-trace analysis and a statistical-honesty framework were deployed to diagnose failures, offering transferable evaluation techniques for low-resource multilingual RAG systems
Industry Insight
- Organizations deploying multilingual RAG systems for low-resource languages should prioritize script selection as a strategic variable, since comprehension gains are heavily script-dependent and not uniform across related languages
- Evaluation pipelines for low-resource domains must go beyond accuracy metrics to include reasoning-trace analysis and statistical-honesty checks, as standard benchmarks may mask fact substitution and confabulation failures
- Investment in model-level reasoning robustness for cross-lingual transfer will yield higher returns than simply expanding retrieval corpora, since the study shows failures originate in model parsing rather than corpus quality
Disclaimer: The above content is generated by AI and is for reference only.