TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
TokenSwap introduces a method for constructing cross-modal inputs by replacing textual concepts with semantically aligned images, interleaving visual and text tokens The modality gap is defined as the performance difference between semantically equivalent textual and multimodal inputs, revealing a pervasive 19.6% average performance drop across 42 MLLMs Reasoning models show significantly smaller modality gaps (10.1%) compared to non-reasoning models (25.5%) Neither prompting strategies nor scal
Analysis
TL;DR
- TokenSwap introduces a method for constructing cross-modal inputs by replacing textual concepts with semantically aligned images, interleaving visual and text tokens
- The modality gap is defined as the performance difference between semantically equivalent textual and multimodal inputs, revealing a pervasive 19.6% average performance drop across 42 MLLMs
- Reasoning models show significantly smaller modality gaps (10.1%) compared to non-reasoning models (25.5%)
- Neither prompting strategies nor scaling training compute alone reliably reduces the modality gap
- Incorporating TokenSwap during training effectively mitigates the gap while preserving strong text-only and vision-language performance
Why It Matters
This research addresses a fundamental reliability concern in multimodal AI systems: models that perform well on text-based benchmarks may degrade significantly when presented with semantically equivalent multimodal inputs. For practitioners building production MLLM applications, this gap represents a hidden failure mode that could lead to inconsistent user experiences. The findings also challenge common assumptions about scaling and prompting as solutions to modality alignment issues.
Technical Details
- TokenSwap Method: Constructs cross-modal test inputs by replacing textual concepts with semantically aligned images, creating sequences where visual tokens are interleaved with text tokens
- TokenSwap-Bench: Transforms existing text-based benchmarks (e.g., MMLU) into image-interleaved counterparts for systematic evaluation
- Empirical Findings: Evaluated across 42 MLLMs, performance decreased by 4.2% to 47.4% (averaging 19.6% +/- 3.3%) when moving from text-only to image-interleaved inputs
- Reasoning Model Advantage: Reasoning models achieved an average gap of 10.1% versus 25.5% for non-reasoning models, suggesting architectural differences in cross-modal consistency
- Mitigation Approach: Training with TokenSwap-generated inputs effectively reduces the modality gap while maintaining strong performance on standard text-only and vision-language benchmarks
Industry Insight
- Benchmark Reliability Concern: Current MLLM evaluations may overestimate real-world performance, as models are predominantly tested on text-only or purely visual inputs rather than mixed modalities
- Reasoning Models as a Design Choice: The smaller modality gap in reasoning models suggests that architectural choices favoring chain-of-thought or extended reasoning may inherently improve cross-modal consistency, informing future model design
- Training Data Strategy: Simply scaling compute or refining prompts is insufficient; practitioners should consider incorporating cross-modal training examples (like TokenSwap) to build more robust multimodal systems, particularly for applications requiring consistent behavior across input modalities
Disclaimer: The above content is generated by AI and is for reference only.