Meet 'Code-as-World': An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs
Code-as-World introduces an Executable World Representation (EWR) paradigm that converts real videos into editable `scene.json` files executable by MuJoCo, treating pixels as evidence rather than ontology An agentic abductive search loop (propose → instantiate → execute → render → verify) recovers physical world programs from video in up to five rounds, outperforming Best-of-5 independent sampling across Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D Verified executable worlds provide
Analysis
TL;DR
- Code-as-World introduces an Executable World Representation (EWR) paradigm that converts real videos into editable
scene.jsonfiles executable by MuJoCo, treating pixels as evidence rather than ontology - An agentic abductive search loop (propose → instantiate → execute → render → verify) recovers physical world programs from video in up to five rounds, outperforming Best-of-5 independent sampling across Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D
- Verified executable worlds provide exact physical labels (mass, friction, gravity, contacts) that raw video lacks, enabling a two-phase training pipeline: SFT on 73,335 image-space QA pairs followed by GRPO reinforcement learning on ~2,573 world-space VQA samples
- Code-as-World-VL-9B achieves 55.4 MRA on QuantiPhy-validation, surpassing Gemini-3.1 Flash (54.8) and leading open-weight baselines by ~15 points; the 4B and 9B checkpoints are released under Apache 2.0
- Limitations include rigid-body-only physics and the model not learning the discovery loop itself — the agentic verification pipeline remains external to the model
Why It Matters
This work addresses a fundamental gap in vision-language models: video models can predict plausible frames but lack grounded physical understanding of mass, contact, and gravity. By converting videos into executable simulations, Code-as-World creates a new class of training data with exact physical labels, pushing VLMs toward genuine physical reasoning rather than statistical pattern matching. For practitioners, it demonstrates a practical path to improving physical-world VQA performance without requiring massive proprietary datasets.
Technical Details
- Executable World Representation (EWR): A triple
p = (C, E, A)— Composition (objects, geometry, mass, friction, gravity), Evolution (initial states, forces, contacts, collisions, duration), and Appearance (camera, lighting, materials, frame rate). Compiles toscene.jsonexecuted in MuJoCo with two interchangeable engines: kinematic animation and physics simulation. - Agentic Inverse-Solving Loop: Uses SAM 3 for instance masks and tracks, VGGT-Omega for depth/camera geometry, and SAM 3D for per-object meshes. Candidate rollouts are projected back into input views and verified across RGB, depth, masks, and trajectories. Structured feedback Δ guides iterative revision over up to K=5 rounds.
- Two-Phase Training: Phase 1 — SFT on 73,335 image-space QA pairs from RefCOCO/+/g, RefCLEF, and GOT-10K covering spatial and kinematic properties. Phase 2 — GRPO on world-space VQA from 1,585 text-driven and 988 video-driven executable worlds, rewarded on scale-normalized numerical accuracy with unit/format penalties. Trained on 8× NVIDIA H100 GPUs.
- Benchmark Results: On QuantiPhy-validation (159 items, MRA over 2S/2D/3S/3D): 4B = 50.6, 9B = 55.4, 27B reasoning = 58.6 vs. Gemini-3.1 Flash 54.8, ChatGPT-5.1 48.4, Qwen3-VL-32B-Instruct 40.2. World-space RL improves pixel-level grounding (9B: RefCOCO 63.7→68.3, GOT-10K 20.1→26.6).
- Deployment: Apache 2.0 checkpoints (VL-4B, VL-9B) fine-tuned from Qwen3.5-4B/9B, BF16 safetensors served via vLLM with OpenAI-compatible
/v1endpoint, 16 sampled frames per video,--max-model-len 4608.
Industry Insight
- The EWR paradigm establishes a blueprint for generating physically grounded training data at scale — any organization with video content can potentially build executable world datasets, reducing reliance on synthetic or manually annotated physical reasoning benchmarks.
- The agentic propose-verify loop demonstrates that iterative refinement with simulation feedback significantly outperforms naive sampling, suggesting broader applicability to other inverse problems in robotics, autonomous systems, and simulation-based AI training.
- The rigid-body limitation and separation of discovery from model learning highlight that current approaches remain narrow; next-generation systems will need to internalize the verification loop and extend to deformable bodies, fluids, and contact-rich manipulation for true general-purpose physical AI.
Disclaimer: The above content is generated by AI and is for reference only.