Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms Khondo：孟加拉语表单文档包分割的多模态基准

TL;DR

• Khondo is the first multimodal benchmark for document packet splitting on Bangladeshi government forms, operating directly on page images rather than OCR text.
• It features bilingual (Bangla-English) content across five concatenation schemes (sequential to fully shuffled) and 14 administrative domains with ground-truth boundaries and page order.
• Zero-shot MLLM evaluation reveals models cluster pages into source documents well but struggle significantly with restoring original page order when shuffled.
• Controlled analyses show explicit page-order instructions are necessary but insufficient, and English packets are ordered more reliably than Bangla packets, establishing page arrangement as the dominant challenge.
• The benchmark isolates page-order reconstruction as a key open problem in vision-based, low-resource document understanding.

Why It Matters

This work addresses a critical gap in document processing for low-resource languages by providing the first vision-native benchmark for document packet splitting, moving beyond OCR-dependent approaches that fail in low-resource settings. The finding that page-order reconstruction is the primary challenge—more so than clustering—offers a clear research direction for improving multimodal document understanding systems, particularly for government and administrative workflows where document packets are common. The bilingual nature of the dataset also highlights language-specific performance differences that must be addressed for equitable AI deployment across languages.

Technical Details

• Khondo operates directly on page images (vision-native) rather than relying on OCR text, making it suitable for low-resource languages where OCR quality is poor.
• The dataset spans five concatenation schemes: sequential, partially shuffled, fully shuffled, and two intermediate variations, across 14 distinct administrative domains (e.g., tax, health, education).
• Ground-truth annotations include document boundaries, domain types, and original page order for every packet, enabling precise evaluation of both clustering and ordering capabilities.
• Zero-shot evaluation of Multimodal Large Language Models (MLLMs) shows strong performance on clustering pages into their source documents but significant degradation when reconstructing original page order after shuffling.
• Controlled experiments vary prompt instructions (explicit vs. implicit page-order guidance) and packet language (Bangla vs. English), revealing that explicit instructions help but are insufficient, and English packets are ordered more reliably than Bangla packets.

Industry Insight

AI practitioners developing document processing systems for government or administrative applications should prioritize page-order reconstruction algorithms over clustering capabilities, as this is the primary bottleneck identified in the Khondo benchmark. Investment in vision-native models that operate directly on page images (rather than OCR text) will be critical for low-resource language support, where OCR quality is often unreliable. The consistent performance gap between English and Bangla packets suggests that language-specific adaptations or training data are necessary for equitable performance across languages, particularly in multilingual government systems.

TL;DR

• 提出首个针对孟加拉政府表单文档包分割的多模态基准Khondo，支持视觉原生与双语处理。
• 覆盖5种拼接方案（顺序至完全打乱）及14个行政领域，提供真实边界、域类型与页序标注。
• MLLMs在聚类页面表现良好，但恢复原始页序困难；语言与指令主要影响排序而非聚类。
• 揭示页序重建是低资源视觉文档理解的关键开放问题，语言为次要但一致影响因素。
• 推动对非英语、低资源场景下文档结构化任务的基准建设与方法论反思。

为什么值得看

该研究填补了低资源语言（孟加拉语）在文档理解领域基准的空白，尤其针对政府表单这一高价值但被忽视的应用场景。其视觉原生设计与多方案拼接设置，为评估多模态大模型在真实复杂文档处理中的鲁棒性提供了新范式，对推动全球AI包容性与本地化落地具有战略意义。

技术解析

Khondo基准构建于孟加拉国政府实际表单数据，采用图像直接输入而非OCR文本，强调视觉语义对齐能力。数据涵盖从简单顺序拼接至完全随机打乱的5种文档包生成策略，确保模型需同时解决“分组”与“排序”双重任务。实验通过零-shot测试主流MLLMs，发现其在识别文档归属（聚类）上有效，但在还原原始页序（排序）上显著退化，尤其当语言为孟加拉语时性能进一步下降。控制变量分析表明：明确指令可辅助排序但无法根本解决问题，而语言差异则系统性地降低排序可靠性，凸显视觉-语言协同建模的不足。

行业启示

• 政府与公共部门数字化进程中，应优先建立本地语言、本地格式的文档处理基准，避免过度依赖英语主导的评估体系。
• 多模态模型开发者需将“页序重建”作为核心能力纳入训练目标，尤其在处理混合语言、无序输入的行政文档时。
• 低资源语言场景下，视觉特征与语言结构的解耦可能成为瓶颈，未来工作应探索跨模态对齐机制以增强顺序感知能力。