OmniDreams

OmniDreams is a generative world model from NVIDIA, mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned driving video in real time. It conditions generation on past frames, the current simulator state and immediate driving actions, and was trained on 21k hours of driving scenarios.

It is aimed at closed-loop evaluation of autonomous driving policies, where the policy's actions update the simulator and determine the next sensor observations. Unlike reconstruction-based neural simulators, which are limited by their captured data, OmniDreams can synthesize unobserved phenomena such as extreme weather and unpredictable agent behaviour. Creator coverage describes multi-camera output from frames, text, maps and trajectories, with weather changes and editing.

NVIDIA reports deploying it in a closed loop with the Alpamayo 1 policy model and the AlpaSim orchestrator. Preliminary results also show a world-action model post-trained from OmniDreams outperforming the VLA-based Alpamayo 1.5 research policy on the Physical AI Autonomous Vehicles NuRec dataset with about 1/5 of the parameters. The report states code and weights are released (GitHub, Hugging Face), but the license and exact links were not in the provided material.