Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning
VIB-ICL introduces an information-theoretic framework using the Information Bottleneck principle to explain and resolve the dichotomy of visual context neglect in multimodal in-context learning Cross-Modal Information Gain (CMIG) is proposed as a metric quantifying the additional mutual information visual context provides beyond textual context The paper proves that visual neglect is the Information Bottleneck-optimal solution when visual information is redundant, yielding a closed-form Attentio
Analysis
TL;DR
- VIB-ICL introduces an information-theoretic framework using the Information Bottleneck principle to explain and resolve the dichotomy of visual context neglect in multimodal in-context learning
- Cross-Modal Information Gain (CMIG) is proposed as a metric quantifying the additional mutual information visual context provides beyond textual context
- The paper proves that visual neglect is the Information Bottleneck-optimal solution when visual information is redundant, yielding a closed-form Attention Reallocation Principle
- VIB-ICL dynamically reallocates attention by estimating CMIG via variational bounds, achieving up to 4.7% accuracy gains and 35% reduction in required demonstrations across five benchmarks
Why It Matters
This work provides the first rigorous theoretical grounding for when and why visual context helps or hurts multimodal ICL, transforming an observed failure mode into an optimizable property. For AI practitioners building multimodal systems, it offers a principled algorithm to adaptively weight visual versus textual demonstrations, directly improving sample efficiency and accuracy. The theoretical bounds also establish clear conditions under which multimodal ICL provably outperforms text-only approaches.
Technical Details
- Cross-Modal Information Gain (CMIG): A novel information-theoretic quantity measuring the additional mutual information that visual demonstrations provide about the target label beyond what textual context alone conveys, defined as the difference in mutual information between multimodal and text-only settings.
- Generalization Bound: The authors derive a theoretical bound showing that multimodal ICL's excess risk over text-only ICL is governed by CMIG, proving that multimodal ICL provably outperforms text-only ICL when visual information is non-redundant.
- Attention Reallocation Principle: A closed-form principle derived from the Information Bottleneck framework that prescribes how visual attention weights should be adaptively adjusted—neglecting visual context is shown to be optimal when visual information is redundant rather than a failure mode.
- VIB-ICL Algorithm: Instantiates the theoretical framework by estimating CMIG via variational bounds and dynamically reallocating attention weights across modalities during inference, enabling adaptive multimodal ICL.
- Empirical Validation: Evaluated on five benchmarks, demonstrating consistent improvements of up to 4.7% accuracy gains and a 35% reduction in the number of required demonstrations, validating the theoretical predictions.
Industry Insight
- The framework reframes visual neglect from a bug to a feature: systems should not force visual attention but instead learn when visual context is informative, enabling more efficient multimodal models that avoid unnecessary computation on redundant modalities.
- The 35% reduction in required demonstrations has direct implications for reducing inference costs and latency in production multimodal systems, where demonstration retrieval and processing are significant overhead.
- The Information Bottleneck perspective opens a new research direction for understanding multi-modal fusion—practitioners should consider information-theoretic measures of modality redundancy when designing few-shot multimodal pipelines.
Disclaimer: The above content is generated by AI and is for reference only.