Black Forest Labs Releases FLUX 3: A Multimodal Flow Model for Image, Video, Audio and Robot Action Prediction Black Forest Labs发布FLUX 3：用于图像、视频、音频和机器人动作预测的多模态流模型

TL;DR

• Black Forest Labs released FLUX 3, a unified multimodal foundation model capable of processing and generating images, video, and audio within a single architecture.
• The model utilizes the "Self-Flow" method, combining flow matching with self-supervised feature reconstruction to align generation and understanding across modalities.
• FLUX 3 Video supports up to 20-second clips with native audio, demonstrating superior human preference rates against competitors like Luma Ray 3.2 and Runway Gen-4.5.
• The same backbone powers FLUX-mimic, a robot policy that operates in under 80 ms on consumer hardware (RTX 5090), highlighting real-time applicability.
• Training is heavily skewed toward video (over 95% of compute) despite audio comprising less than 0.5% of tokens, emphasizing the complexity of spatiotemporal dynamics.

Why It Matters

FLUX 3 represents a significant shift from unimodal or loosely coupled multimodal systems to a deeply integrated architecture where modalities constrain and inform each other during training. This approach addresses the fundamental limitation that no single modality provides a complete description of reality, offering practitioners a more robust tool for complex generative tasks. For the industry, the ability to generate synchronized video and audio with high fidelity, while simultaneously enabling low-latency robotic control, bridges the gap between creative media production and physical world interaction.

Technical Details

• Architecture & Method: Built on the Self-Flow framework, which merges flow matching objectives with self-supervised feature reconstruction. It uses a SiT-XL/2 backbone with per-token timestep conditioning, trained with a 25% per-token mask ratio and self-distillation from an EMA teacher (layer 20) to a student (layer 8).
• Multimodal Integration: Unlike previous models, FLUX 3 trains jointly on images, videos, and audio. The team posits that audio reveals causal relationships and video exposes physical dynamics, allowing the model to learn consistent physical laws (e.g., sound matching impact, motion obeying mass).
• Video Capabilities: Supports text-to-video, image-to-video, video-to-video, keyframe-to-video, and generative video-audio continuation. It generates clips up to 20 seconds long in a single pass with native audio synchronization.
• Performance Benchmarks: In human preference tests for 10-second 720p clips with audio, FLUX 3 outperformed Luma Ray 3.2 (93% preference) and Runway Gen-4.5 (77% preference), showing competitive results against Grok Imagine Video and Kling v3 Pro.
• Robotics Application: The backbone drives FLUX-mimic, a robot policy capable of running inference in under 80 milliseconds on a single RTX 5090 GPU, demonstrating efficient real-world deployment potential.

Industry Insight

The heavy computational bias toward video (95% of training resources) suggests that future advancements in multimodal AI will likely prioritize spatiotemporal coherence over simple audio-visual alignment. Developers should anticipate a trend toward unified backbones that can handle diverse inputs without separate specialized modules, reducing integration complexity. Furthermore, the successful application of this large-scale generative model to low-latency robotics indicates that foundation models are becoming viable for real-time control loops, opening new avenues for embodied AI research and commercial applications.

TL;DR

• Black Forest Labs 发布 FLUX 3，首个在单一架构中联合学习图像、视频和音频的多模态基础模型。
• 采用 Self-Flow 方法，结合流匹配目标与自监督特征重建，实现多模态生成与理解的对齐。
• FLUX 3 Video 支持单次生成长达 20 秒的视频并附带原生音频，涵盖多种输入模式及多语言对话。
• 在人类偏好测试中，FLUX 3 显著优于 Luma Ray 3.2 (93%) 和 Runway Gen-4.5 (77%)，但在部分模型上优势不明显。
• 同一骨干网络驱动机器人策略 FLUX-mimic，可在单张 RTX 5090 上以低于 80ms 的延迟运行。

为什么值得看

这篇文章展示了多模态大模型从单一模态向统一架构演进的关键一步，证明了通过联合训练不同模态数据可以相互约束并提升整体表现。对于 AI 从业者而言，了解 Self-Flow 技术及多模态对齐的实现细节，有助于把握下一代通用人工智能的基础设施发展趋势。

技术解析

• 架构与方法：基于 Self-Flow 方法，将流匹配（flow matching）目标与自监督特征重建目标结合。使用 SiT-XL/2 架构，支持每 token 的时间步条件注入，训练时采用 25% 的 per-token 掩码比率，并通过 EMA 教师模型进行自蒸馏（从第 20 层到第 8 层）。
• 多模态统一训练：模型在图像、视频和音频数据上同时训练，利用不同模态对同一底层现实的“有损投影”特性，使声音必须匹配冲击，运动必须遵循质量，从而实现模态间的相互约束。
• 视频生成能力：支持文本转视频、图像转视频、参考视频转视频、关键帧转视频以及视频音频延续生成。能够生成高达 20 秒的单次视频片段，并具备原生音频生成能力，特别擅长人类面部表情和物理事件与声音的关联。
• 资源分配与扩展：视频预测消耗了超过 95% 的训练算力，而音频仅占不到 0.5% 的 token。BFL 表示相比之前的研究模型，FLUX 3 显著增加了算力和数据资源的投入。

行业启示

• 多模态融合是必然趋势：单一模态无法完整描述世界，未来模型将更倾向于在统一架构下联合处理视觉、听觉和动作信息，以实现更连贯的物理世界模拟。
• 推理效率与硬件优化：FLUX-mimic 在消费级高端显卡（RTX 5090）上实现低延迟机器人控制，表明多模态大模型正逐步具备实时交互和边缘部署的潜力，降低了机器人智能化的硬件门槛。
• 竞争格局动态变化：虽然 FLUX 3 在多项基准中领先，但与 Gemini Omni Flash 等模型的对比接近抛硬币，说明多模态视频生成领域的竞争依然激烈，技术迭代迅速，需持续关注后续开源权重发布及性能突破。