Post-Train NVIDIA Cosmos 3 Edge for On-Device Robot Control
NVIDIA Cosmos 3 Edge is a 4B omni-model with a 2B Nemotron-based reasoner, designed to run on-device on NVIDIA Jetson Thor for real-time robot control Post-training transforms the world foundation model into a manipulation policy using the Cosmos3-DROID dataset (76k trajectories, ~350 hours across 86 tasks) The policy achieves 22.9% success rate in closed-loop RoboLab tasks, generating action chunks in ~1.53 seconds at 15 Hz on Jetson AGX Thor T5000 Training requires significant distributed comp
Analysis
TL;DR
- NVIDIA Cosmos 3 Edge is a 4B omni-model with a 2B Nemotron-based reasoner, designed to run on-device on NVIDIA Jetson Thor for real-time robot control
- Post-training transforms the world foundation model into a manipulation policy using the Cosmos3-DROID dataset (76k trajectories, ~350 hours across 86 tasks)
- The policy achieves 22.9% success rate in closed-loop RoboLab tasks, generating action chunks in ~1.53 seconds at 15 Hz on Jetson AGX Thor T5000
- Training requires significant distributed compute: 64 nodes of 4× GB200 for 60K iterations (~17.4K GB200-hours), not a simple single-GPU fine-tune
- The approach enables continuous on-device streaming by generating action chunks and replanning after each inference cycle rather than after every observation
Why It Matters
This represents a significant step toward deploying capable world foundation models directly on robotic hardware without cloud dependency, addressing the critical constraints of device memory and control latency that have historically limited on-robot AI. For AI practitioners working in robotics, it demonstrates that post-training a 4B-parameter model can yield practical real-time policies, bridging the gap between large-scale world models and edge deployment. The open framework and released checkpoints lower the barrier for researchers to build upon this foundation.
Technical Details
- Model Architecture: Cosmos 3 Edge is a 4B omni-model featuring a 2B NVIDIA Nemotron-based reasoner, pretrained on the same physical-world data as Cosmos 3 Nano and Super, with freshly initialized action heads (encoder + decoding MLP + embedding tokens) using a 5× learning rate multiplier
- Training Configuration: Joint position action space (8-D absolute: 7 joints + gripper), 3-camera canvas (wrist 360×640 over two exterior views 180×320 each → 540×640), action chunk of 32 future actions at 15 Hz, vision flow-matching loss weighted to match action loss, learning rate 2e-4 with global batch 8192
- Dataset: Cosmos3-DROID contains 76k successful teleoperated trajectories (~350 hours, 86 tasks, 564 scenes) collected with Franka Panda arm and Robotiq gripper, packaged in LeRobotDataset v3.0 format at 640×360 resolution, with filtering of idle/non-task frames and data augmentation via random cropping, rescaling, and color jitter
- Inference Performance: On Jetson AGX Thor T5000, each action chunk generates in ~1.53 seconds at 640×540 resolution and 15 Hz, covering ~2.13 seconds of robot motion, enabling continuous streaming with receding-horizon control and replanning after each inference cycle
- Compute Requirements: Distributed post-training across 64 nodes with 4× GB200 each for 60K iterations (~68 hours, ~17.4K GB200-hours), using CUDA 13.0 and NGC 26.06-py3, with long-cosine decay schedule and HSDP 32×8 parallelism
Industry Insight
- The ability to run a 4B-parameter world model entirely on edge hardware (Jetson Thor) without offloading to data-center GPUs marks a turning point for autonomous robotics, enabling low-latency, privacy-preserving, and always-available robot intelligence
- The modest 22.9% closed-loop success rate highlights that while on-device deployment is now feasible, significant room remains for improvement in policy robustness—researchers should focus on better sim-to-real transfer, richer embodiment support, and improved post-training techniques
- The open release of the framework, checkpoints, and dataset on HuggingFace democratizes access to world-model-based robot policies, but the substantial distributed training compute requirement (~17.4K GB200-hours) means only well-resourced organizations can replicate the full post-training pipeline, creating a potential capability gap in the robotics community
Disclaimer: The above content is generated by AI and is for reference only.