DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
DocHop is a new benchmark for evaluating integrated chart-context reasoning in information-dense document-style images, addressing a gap where existing benchmarks evaluate charts and documents in isolation The benchmark uses a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, producing 2,074 examples across six task categories A significant performance gap exists between humans and MLLMs: human annotators achieve over 90% accuracy while the best mod
Analysis
TL;DR
- DocHop is a new benchmark for evaluating integrated chart-context reasoning in information-dense document-style images, addressing a gap where existing benchmarks evaluate charts and documents in isolation
- The benchmark uses a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, producing 2,074 examples across six task categories
- A significant performance gap exists between humans and MLLMs: human annotators achieve over 90% accuracy while the best model reaches only 62.83%
- Reasoning-enhanced models show consistent improvements, but performance degrades as reasoning complexity increases
- DocHop requires models to resolve target entities from textual context before aggregating evidence across multiple charts, testing true multi-hop compositional reasoning
Why It Matters
This benchmark addresses a critical gap in multimodal AI evaluation by testing whether models can integrate textual narratives with visual chart data in realistic document-style images, rather than evaluating these modalities in isolation. For AI practitioners, it highlights that current MLLMs still struggle significantly with multi-hop reasoning tasks that require cross-modal evidence aggregation, suggesting this remains an open challenge for production-grade document understanding systems.
Technical Details
- Benchmark Design: DocHop evaluates integrated chart-context reasoning where document narratives specify multi-step compositional constraints and charts provide corresponding data values, requiring models to resolve semantic reference labels from context before aggregating cross-chart evidence
- Data Generation: Constructed via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density parameters, ensuring systematic and scalable benchmark creation
- Dataset Scale: 2,074 examples spanning six distinct task categories, designed to test out-of-domain generalization in information-dense document-style images
- Evaluation Results: Human annotators achieve >90% accuracy while the best proprietary/open-source MLLM reaches only 62.83%, with reasoning-enhanced models showing improvement but consistent degradation as complexity increases
- Task Categories: Six categories covering different forms of multi-hop compositional reasoning across chart-context pairs in document layouts
Industry Insight
- The 27+ percentage point gap between human and model performance on multi-hop document reasoning suggests significant room for improvement in multimodal architectures, particularly in cross-modal entity resolution and evidence aggregation
- The consistent degradation with increasing reasoning complexity indicates that current MLLMs lack robust compositional generalization, making reasoning-enhancement techniques a promising direction for future model development
- Organizations building document understanding systems should prioritize out-of-domain multi-hop reasoning capabilities and consider DocHop as an evaluation standard before deploying in production environments
Disclaimer: The above content is generated by AI and is for reference only.