NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection
NepOOC is the first publicly available Nepali-dominant multilingual benchmark for out-of-context (OOC) misinformation detection, containing 1,090 image-caption pairs with high inter-annotator agreement (kappa = 0.84) OOC misinformation pairs authentic images with misleading captions, making detection a multimodal alignment problem rather than traditional image forensics A text-only mBERT model achieves 94.65±0.20% Macro-F1, statistically equivalent to the best multimodal system (ResNet-50+mBERT)
Analysis
TL;DR
- NepOOC is the first publicly available Nepali-dominant multilingual benchmark for out-of-context (OOC) misinformation detection, containing 1,090 image-caption pairs with high inter-annotator agreement (kappa = 0.84)
- OOC misinformation pairs authentic images with misleading captions, making detection a multimodal alignment problem rather than traditional image forensics
- A text-only mBERT model achieves 94.65±0.20% Macro-F1, statistically equivalent to the best multimodal system (ResNet-50+mBERT), indicating caption semantics alone are sufficient at current dataset scale
- Image-only models perform near chance (33-50%), confirming that visual content alone cannot reliably detect OOC mismatches
- Dataset expansion, not architectural sophistication or regional specialization, is identified as the most direct path to performance gains
Why It Matters
This work addresses a critical gap in misinformation research by providing the first benchmark for Nepali, a language with significant misinformation prevalence but no public evaluation resources. For AI practitioners, the finding that text-only models match multimodal systems challenges the assumption that more modalities always yield better detection, suggesting resource-efficient alternatives for low-resource language settings.
Technical Details
- Benchmark composition: 1,090 image-caption pairs (545 pristine, 545 OOC) annotated across five typologies: fabricated, miscaptioned, temporal mismatch, geographic mismatch, and identity mismatch
- Model evaluation: Five multimodal architectures tested alongside text-only and image-only baselines, with mBERT (multilingual BERT) as the text encoder and ResNet-50 as the visual encoder
- Performance results: Text-only mBERT achieved 94.65±0.20% Macro-F1; best multimodal (ResNet-50+mBERT) matched at 94.65±0.20%; image-only models scored 33-50% (near chance)
- Statistical validation: McNemar median test yielded p = 1.000, with 0/5 seeds showing significant differences at alpha = 0.05, confirming equivalence between text-only and multimodal approaches
- Annotation quality: Inter-annotator agreement kappa = 0.84, indicating strong consistency in labeling
Industry Insight
- Organizations combating misinformation in low-resource languages should prioritize dataset creation and expansion over investing in complex multimodal architectures, as text signals dominate detection capability
- The near-chance performance of image-only models validates the conceptual framing of OOC detection as an alignment problem, redirecting research efforts toward cross-modal reasoning rather than visual forensics
- This benchmark establishes a template for building multilingual misinformation datasets, suggesting that regional specialization may be less critical than scale and typology diversity in driving performance improvements
Disclaimer: The above content is generated by AI and is for reference only.