Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio
The authors introduce the Fusion Embedding family, creating a unified multi-modal space for text, image, video, and audio by adding audio capabilities to existing frozen vision-language bases. Generation 1 utilizes a lightweight 16.4M-parameter connector between a frozen audio encoder and the frozen base, while Generation 2 employs modality-gated deep adapters (44.2M parameters) that remain inactive for non-audio inputs. Aligning audio exclusively to text enables emergent audio-to-image and audi
Analysis
TL;DR
- The authors introduce the Fusion Embedding family, creating a unified multi-modal space for text, image, video, and audio by adding audio capabilities to existing frozen vision-language bases.
- Generation 1 utilizes a lightweight 16.4M-parameter connector between a frozen audio encoder and the frozen base, while Generation 2 employs modality-gated deep adapters (44.2M parameters) that remain inactive for non-audio inputs.
- Aligning audio exclusively to text enables emergent audio-to-image and audio-to-video retrieval without requiring any paired audio-visual training data.
- The models are highly efficient, training in hours on a single GPU, with both weights and code openly released.
- Controlled experiments reveal that simplifying training captions, using stronger audio towers, or widening connectors actually degrades retrieval performance, highlighting specific design constraints.
Why It Matters
This approach significantly lowers the barrier to entry for building comprehensive multi-modal search systems by leveraging pre-existing vision-language embeddings rather than training from scratch. It demonstrates that cross-modal alignment can emerge indirectly through a shared textual anchor, reducing the need for expensive and scarce paired audio-visual datasets. For practitioners, this offers a scalable, compute-efficient pathway to integrate audio into existing multi-modal infrastructures.
Technical Details
- Architecture: The method builds upon frozen vision-language embedding bases. It introduces an audio tower and connects it to the base via either a simple connector (Gen 1, 16.4M params) or modality-gated deep adapters (Gen 2, 44.2M params).
- Modality Gating: In Generation 2, the adapters are strictly gated; they do not execute for text, image, or video inputs, ensuring bit-for-bit identical outputs to the base model for those modalities.
- Training Strategy: Only the connector/adapters are trained; the base parameters remain frozen. Audio is aligned solely to text representations.
- Emergent Capabilities: Despite training only on audio-text pairs, the system supports audio-image and audio-video retrieval due to the underlying binding of text, image, and video in the frozen base.
- Negative Results: The study maps the design space, finding that LLM-based caption rewriting, substituting stronger audio encoders, and increasing connector width all negatively impact retrieval accuracy.
Industry Insight
- Cost-Efficient Multi-Modal Expansion: Organizations can extend existing text/image/video search pipelines to include audio with minimal computational overhead and no need for new paired visual-audio data collection.
- Design Sensitivity: The counter-intuitive finding that simpler components outperform more complex ones (like wider connectors or better audio towers) suggests that over-engineering the integration layer may harm generalization in unified embedding spaces.
- Standardization Potential: This modular approach of freezing a strong base and plugging in specialized adapters could become a standard pattern for integrating new modalities into established foundation models.
Disclaimer: The above content is generated by AI and is for reference only.