XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering
XHotpotQA is a new benchmark for cross-lingual knowledge composition in multi-hop question answering, addressing a gap where existing multilingual benchmarks translate entire examples into one language, thereby hiding failures at language boundaries The benchmark models each instance as an evidence-dependency graph with explicit language assignments for questions, bridge evidence, answer-bearing evidence, and distractors The dataset contains 15,661 training and 7,405 validation instances with se
Analysis
TL;DR
- XHotpotQA is a new benchmark for cross-lingual knowledge composition in multi-hop question answering, addressing a gap where existing multilingual benchmarks translate entire examples into one language, thereby hiding failures at language boundaries
- The benchmark models each instance as an evidence-dependency graph with explicit language assignments for questions, bridge evidence, answer-bearing evidence, and distractors
- The dataset contains 15,661 training and 7,405 validation instances with sentence-level support supervision and supplied distractors
- Full question-evidence language mismatch causes 10.25–15.79 lower Unicode-aware answer F1, while different-script evidence causes deficits of 11.98–23.70 points
- The benchmark provides role-aware diagnostics and modular evaluation for systems that must integrate evidence across languages
Why It Matters
This benchmark directly addresses a critical blind spot in multilingual NLP evaluation: the failure modes that occur specifically at language boundaries within reasoning chains. For AI practitioners building multilingual QA systems, XHotpotQA offers a controlled testbed to diagnose and improve cross-lingual knowledge composition rather than relying on benchmarks that mask these failures through full translation.
Technical Details
- Each instance is modeled as an evidence-dependency graph with explicit language assignments for the question, bridge evidence, answer-bearing evidence, and distractors, enabling fine-grained analysis of cross-lingual failure points
- The dataset contains 15,661 training and 7,405 validation instances with sentence-level support supervision and supplied distractors, providing a controlled environment for evaluation
- Validation statistics show 99.81% of items cross the question-to-gold-evidence language interface and 95.60% use gold paragraphs in different languages, ensuring substantial cross-lingual coverage
- Empirical results across three reader artifacts demonstrate that full question-evidence mismatch reduces Unicode-aware answer F1 by 10.25–15.79 points compared to partial alignment, and different-script evidence causes deficits of 11.98–23.70 points
- The adapted-selector component shows substantially smaller condition-associated deficits (1.71 and 1.78 points) compared to readers, highlighting the relative robustness of evidence selection versus composition under cross-lingual conditions
Industry Insight
- Multilingual QA systems should invest in cross-lingual evidence selection and composition modules rather than relying on translation-based approaches, as language boundary failures represent a significant and under-evaluated source of error
- The modular evaluation framework of XHotpotQA can guide targeted improvements in knowledge-intensive NLP pipelines, particularly for systems operating in low-resource or mixed-script environments
- Benchmark designers and practitioners should consider role-aware diagnostics when evaluating multilingual systems, as aggregate metrics may obscure critical failure modes at specific points in the reasoning chain
Disclaimer: The above content is generated by AI and is for reference only.