Compositional Failure in Audio-Visual LLMs: Late-Layer Prior Dominance Under Cross-modal Conflict
Audio-visual LLMs exhibit a failure mode called "prior dominance" where they commit to internally preferred answer patterns rather than resolving cross-modal conflicts between audio and video inputs VideoLLaMA 2-7B-AV remains near chance performance on AVHBench's exact-string Yes/No subset across three alignment configurations, despite substantial shifts in output priors InternVideo2 suffers a 32.3% accuracy drop under cross-modal conflict conditions, with an additional 17.3% instruction-followi
Analysis
TL;DR
- Audio-visual LLMs exhibit a failure mode called "prior dominance" where they commit to internally preferred answer patterns rather than resolving cross-modal conflicts between audio and video inputs
- VideoLLaMA 2-7B-AV remains near chance performance on AVHBench's exact-string Yes/No subset across three alignment configurations, despite substantial shifts in output priors
- InternVideo2 suffers a 32.3% accuracy drop under cross-modal conflict conditions, with an additional 17.3% instruction-following failure rate
- Mechanistic interpretability analysis reveals that commitment to answers concentrates at layers 25.5 ± 1, indicating late-layer dominance in decision-making
- Stronger temporal alignment modifies answer bias but fails to improve compositional conflict resolution, suggesting a fundamental architectural limitation
Why It Matters
This research exposes a critical vulnerability in multimodal AI systems that process both audio and video: when sensory inputs conflict, models default to internal priors rather than performing genuine cross-modal reasoning. For practitioners building AV-LLM applications, this means current systems may appear competent on aligned data but fail catastrophically in real-world scenarios where audio-visual mismatches occur, such as in video analysis, robotics, or interactive AI systems.
Technical Details
- Benchmark and Models: Evaluated on AVHBench using VideoLLaMA 2-7B-AV (three alignment configurations) and off-the-shelf InternVideo2, testing exact-string Yes/No responses under cross-modal conflict conditions
- Prior Dominance Mechanism: Late-layer commitment (concentrated at 25.5 ± 1 layers) to internally preferred answer patterns that are weakly grounded in the actual conflicting inputs, identified through mechanistic interpretability analysis
- Temporal Alignment Findings: Strengthening temporal alignment between audio and video streams shifts answer bias but does not resolve compositional conflict, indicating the problem is architectural rather than purely alignment-related
- Quantified Failures: 32.3% accuracy decrease under cross-modal conflict for InternVideo2, with 17.3% instruction-following failure rate; VideoLLaMA 2-7B-AV remains near chance despite output prior shifts
- Reproducibility: Code and data for mechanistic audit and behavioral evaluations are publicly available
Industry Insight
- Developers of multimodal AI systems should treat cross-modal conflict scenarios as essential stress tests, not edge cases, since real-world deployments will inevitably encounter audio-visual mismatches that expose prior dominance failures
- The finding that late-layer commitment drives this failure suggests architectural interventions—such as intermediate fusion mechanisms or layer-aware conflict resolution—may be more effective than simply improving temporal alignment
- Researchers should prioritize mechanistic interpretability audits for multimodal models to identify where and how conflicting evidence is (or is not) being integrated before final predictions are made
Disclaimer: The above content is generated by AI and is for reference only.