DreamX

DreamX is a research model from KAIST's CVLab (project page cvlab-kaist.github.io) that repurposes a pretrained video diffusion model as a universal latent action learner. One diffusion backbone acts as both encoder and decoder. Learnable queries read compact latent actions from video (4 tokens per frame, roughly 300× compression). The decoder then reconstructs the future video from the first frame plus those latents, so the latents encode motion rather than appearance.

The same latent actions are reused for several tasks: video transfer (encode a source video, decode from a new first frame, across scenes, robots and humans), action transfer (decode into robot actions from the robot's initial state, with no paired human–robot data), and a compact latent policy that predicts 32-dimensional codes which an action DiT turns into robot actions. It was pretrained on 2.19k hours of egocentric human and robot video, and human clips need no action labels. The authors report leading results on LARYBench for action regression and classification, and demonstrate open-loop human-to-robot transfer on a bimanual YAM robot.

DreamX is a research project, not a product. The project page says code is coming soon, according to a creator summary. No released weights, license, demo or API were found in the provided material.