Which Source Wins? Task-Dependent Reliance in Vision-Language Models
Vision-language models dynamically reallocate modality reliance when image and text conflict, with the direction of shift depending on task type rather than being fixed On arithmetic benchmarks (GSM8K, SVAMP), five of six open-weight VLMs shift more strongly away from degraded text than degraded images, favoring visual input On ChartQA-Conflict (a new benchmark of 229 chart-report conflicts), all six models reverse this pattern, shifting more strongly away from degraded visual sources The revers
Analysis
TL;DR
- Vision-language models dynamically reallocate modality reliance when image and text conflict, with the direction of shift depending on task type rather than being fixed
- On arithmetic benchmarks (GSM8K, SVAMP), five of six open-weight VLMs shift more strongly away from degraded text than degraded images, favoring visual input
- On ChartQA-Conflict (a new benchmark of 229 chart-report conflicts), all six models reverse this pattern, shifting more strongly away from degraded visual sources
- The reversal persists after calibrating for unimodal accuracy loss and when replacing charts with plain table images, indicating it is task-structure-driven rather than modality-quality-driven
- Frontier API models GPT-5.6-Luna and Gemini-3.5-Flash replicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction
Why It Matters
This research reveals that VLM modality reliance is not a static property but a task-dependent phenomenon, which has direct implications for how practitioners should evaluate and deploy multimodal systems in real-world scenarios where inputs may be degraded or conflicting. Understanding these shifts is critical for building robust VLM applications in domains like healthcare, finance, and autonomous systems where visual and textual evidence may not always align.
Technical Details
- The authors introduce a controlled experimental setup where either the image or text is degraded across four legibility levels while the other modality remains clean, enabling precise measurement of modality preference shifts
- They construct cross-modal conflicts on GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, creating situations where the two sources support different answers
- A new benchmark, ChartQA-Conflict, is introduced containing 229 manually reviewed chart-report conflicts with matched chart and table-image representations
- Evaluation uses both generated answers and a length-normalized conditional log-likelihood margin across six open-weight VLMs, with behavioral replication tested on GPT-5.6-Luna and Gemini-3.5-Flash
- Robustness checks include calibrating for unimodal accuracy loss and substituting charts with plain table images to isolate task-structure effects from modality-specific factors
Industry Insight
- VLM evaluation pipelines should incorporate conflict-based benchmarks like ChartQA-Conflict to uncover hidden modality biases that standard accuracy metrics may miss, ensuring more reliable deployment in production
- Practitioners building multimodal systems should anticipate that model behavior will shift unpredictably when modalities degrade unevenly, and should implement fallback mechanisms or uncertainty quantification for such scenarios
- The task-dependent nature of modality reliance suggests that model cards and documentation should specify not just overall performance but also modality robustness profiles across different evidence structures to guide appropriate use cases
Disclaimer: The above content is generated by AI and is for reference only.