Flow-JEPA: Flow Matching for Robust Latent Dynamics in JEPA World Models
Flow-JEPA replaces deterministic autoregressive dynamics in JEPA world models with conditional flow matching for robust latent trajectory prediction The model uses a Gaussian distribution as the flow source, exposing the vector field to perturbed latent trajectories during training to improve noise robustness F-JEPA improves mean success rate from 86% to 92% under clean observations and from 67% to 86% under noisy conditions The approach retains the reconstruction-free nature of JEPA while shift
Analysis
TL;DR
- Flow-JEPA replaces deterministic autoregressive dynamics in JEPA world models with conditional flow matching for robust latent trajectory prediction
- The model uses a Gaussian distribution as the flow source, exposing the vector field to perturbed latent trajectories during training to improve noise robustness
- F-JEPA improves mean success rate from 86% to 92% under clean observations and from 67% to 86% under noisy conditions
- The approach retains the reconstruction-free nature of JEPA while shifting from point-wise transition regression to stochastic trajectory-level prediction
- This work addresses error accumulation and visual perturbation sensitivity inherent in LeWorldModel's autoregressive one-step transition approach
Why It Matters
This research directly addresses two critical failure modes in latent world models—error accumulation across autoregressive steps and sensitivity to irrelevant visual noise—which are major bottlenecks for deploying JEPA-based systems in real-world robotics and control tasks. By demonstrating that conditional flow matching can significantly improve both clean and noisy performance, it opens a practical path toward more robust world models without sacrificing the efficiency gains of reconstruction-free architectures.
Technical Details
- Architecture: Flow-JEPA (F-JEPA) is built on the Joint-Embedding Predictive Architecture (JEPA) framework, specifically extending LeWorldModel (LeWM), which performs reconstruction-free latent world modeling directly from pixels
- Core Innovation: Replaces deterministic autoregressive one-step transition prediction with conditional flow matching that jointly generates entire sequences of future latent states conditioned on current observations and actions
- Training Mechanism: Uses a Gaussian distribution as the flow source; the vector field is exposed to perturbed latent trajectories during training, learning to transport them toward clean future representations
- Performance Gains: Mean success rate improves from 86% to 92% under clean conditions and from 67% to 86% under noisy conditions, demonstrating substantial robustness gains
- Key Design Choice: Maintains the reconstruction-free JEPA paradigm while shifting the prediction objective from point-wise regression to stochastic trajectory-level flow matching
Industry Insight
- The significant robustness improvement under noisy conditions (19-point gain) suggests that flow matching-based dynamics modeling should be prioritized for real-world deployment scenarios where visual perturbations are unavoidable, such as robotics and autonomous systems
- The trajectory-level prediction approach could become a standard alternative to autoregressive dynamics in world model research, potentially influencing the direction of JEPA-family architectures beyond the current LeWM variant
- Practitioners working on latent world models should consider conditional flow matching as a drop-in improvement for existing autoregressive predictors, particularly when error accumulation over long horizons is a limiting factor
Disclaimer: The above content is generated by AI and is for reference only.