Do MLLMs Really Understand Low-Resource Khmer Documents? A Pilot Study on Khmer Document VQA
Pilot diagnostic evaluation of open MLLMs on Khmer document images reveals significant gaps in low-resource, non-Latin script understanding Qwen3-VL-8B achieves 51.9% overall accuracy on direct image-based prompting, outperforming smaller Qwen-VL variants External OCR assistance substantially boosts performance: Tesseract reaches 61.9% and PaddleOCR reaches 61.6% Khmer-script and mixed-script answers remain substantially more difficult than English and numeric fields Current MLLMs can handle vis
Analysis
TL;DR
- Pilot diagnostic evaluation of open MLLMs on Khmer document images reveals significant gaps in low-resource, non-Latin script understanding
- Qwen3-VL-8B achieves 51.9% overall accuracy on direct image-based prompting, outperforming smaller Qwen-VL variants
- External OCR assistance substantially boosts performance: Tesseract reaches 61.9% and PaddleOCR reaches 61.6%
- Khmer-script and mixed-script answers remain substantially more difficult than English and numeric fields
- Current MLLMs can handle visually clear English and structured numeric content, but reliable native Khmer document understanding remains an open challenge
Why It Matters
This study highlights a critical gap in the multimodal AI landscape: while MLLMs have made remarkable progress on high-resource languages and Latin scripts, their performance degrades significantly on low-resource, non-Latin writing systems like Khmer. For AI practitioners building document understanding systems for Southeast Asian markets or multilingual applications, these findings serve as a cautionary signal that off-the-shelf models cannot be assumed to generalize across scripts and languages without targeted evaluation and potential OCR hybrid approaches.
Technical Details
- Dataset: Evaluation subset constructed from the KH-FUNSD collection, covering invoices, receipts, quotations, and other business forms with questions in both English and Khmer, and answers in original English, Khmer, mixed-script, or numeric forms
- Models Evaluated: Representative open Qwen-VL models, with Qwen3-VL-8B as the primary focus for direct prompting, parser-assisted, and external OCR-assisted configurations
- Prompting Strategies: Three configurations compared — direct image-based prompting, parser-assisted, and external OCR-assisted (using Tesseract and PaddleOCR)
- Performance Results: Direct Qwen3-VL-8B achieves 51.9% overall accuracy; external OCR-assisted configurations reach 61.9% (Tesseract) and 61.6% (PaddleOCR); Khmer-script answers show notably lower accuracy compared to English and numeric fields
- Study Scope: Positioned as a pilot diagnostic evaluation rather than a full public benchmark, focusing on examining capabilities and failure modes of existing models
Industry Insight
- Organizations deploying MLLM-based document processing in Southeast Asian contexts should not rely on direct vision-only pipelines for Khmer or similar low-resource scripts; hybrid OCR-assisted approaches yield measurably better results and should be considered standard practice
- The performance gap between English/numeric fields and Khmer-script answers underscores the need for targeted data collection and model training on non-Latin document datasets, presenting an opportunity for both academic research and commercial investment in underrepresented language AI
- This diagnostic framework — evaluating failure modes rather than proposing a new benchmark — offers a replicable methodology for assessing MLLM reliability across other low-resource languages and scripts, encouraging the community to adopt similar diagnostic studies before deploying models in production multilingual environments
Disclaimer: The above content is generated by AI and is for reference only.