CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
CAT-GS addresses three coupled failure modes in multimodal neural network training: modality imbalance, unstable gating, and fusion interference The method operates during backpropagation without modifying model architectures, fusion modules, or task losses CAT-GS uses temperature scaling and EMA smoothing to calibrate teacher-derived reliability, then applies a margin-thresholded policy switching between warm-up dropout, weak-modality prioritization, and weak-biased blending Gradient stabilizat
Analysis
TL;DR
- CAT-GS addresses three coupled failure modes in multimodal neural network training: modality imbalance, unstable gating, and fusion interference
- The method operates during backpropagation without modifying model architectures, fusion modules, or task losses
- CAT-GS uses temperature scaling and EMA smoothing to calibrate teacher-derived reliability, then applies a margin-thresholded policy switching between warm-up dropout, weak-modality prioritization, and weak-biased blending
- Gradient stabilization is achieved through capped gradient-budget renormalization and fusion-only PCGrad to reduce destructive cross-modal interference
- Evaluated across six benchmarks (CREMA-D, AV-MNIST, VGGSound, UR-FUNNY, CG-MNIST, AVE, CMU-MOSI), CAT-GS matches or improves accuracy over strong baselines like OGM-GE, G²D, and UMT
Why It Matters
Multimodal learning is increasingly central to AI systems, yet training instability remains a persistent bottleneck that limits performance and reproducibility. CAT-GS offers a plug-and-play optimization controller that can be integrated into existing multimodal pipelines without architectural changes, making it immediately actionable for practitioners. The work also provides a principled framework for understanding and diagnosing the coupled failure modes that arise during end-to-end multimodal training.
Technical Details
- Problem framing: Identifies three coupled failure modes—modality imbalance (one branch dominates gradients), unstable gating (noisy confidence cues cause erratic modality selection), and fusion interference (conflicting gradients at shared fusion layers)
- Calibration mechanism: Uses temperature scaling and exponential moving average (EMA) smoothing to calibrate teacher-derived reliability scores, producing stable confidence estimates for gating decisions
- Margin-thresholded gating policy: Dynamically switches between three regimes—warm-up dropout (early training), weak-modality prioritization (underperforming modalities get more attention), and weak-biased blending (balanced fusion)—based on calibrated reliability margins
- Gradient stabilization: Employs capped gradient-budget renormalization to prevent gradient explosion under aggressive gating, and applies PCGrad (Project Conflicting Gradients) exclusively at the fusion layer to eliminate destructive cross-modal gradient interference
- Evaluation: Tested on audio-visual benchmarks (CREMA-D, AV-MNIST, VGGSound), tri-modal setting (UR-FUNNY), synthetic data (CG-MNIST), and cross-domain benchmarks (AVE, CMU-MOSI), comparing against OGM-GE, G²D, and UMT baselines
Industry Insight
- CAT-GS's architecture-agnostic design means it can be adopted as a drop-in optimization layer in existing multimodal systems, reducing the barrier to implementing robust multimodal training without costly architectural redesign
- The explicit modeling of coupled failure modes provides a diagnostic lens for practitioners experiencing training instability in their own multimodal pipelines—modality imbalance, gating noise, and fusion conflict should be checked as interrelated rather than isolated issues
- As multimodal models scale to three or more modalities, gradient interference at fusion layers will become increasingly severe; fusion-only PCGrad offers a targeted, computationally efficient mitigation strategy worth adopting in multi-modal architectures
Disclaimer: The above content is generated by AI and is for reference only.