PixelUMM

PixelUMM is a research model from NVIDIA (Toronto AI Lab) that handles image and video understanding and generation with a single decoder-only Transformer operating on raw pixels. It uses no VAE and no separate vision encoder. Images are split into 16×16 patches and videos into 4-frame tubelets. The backbone is Qwen3-8B, with separate understanding and generation experts that share one self-attention across text, clean pixels and noisy pixels.

The project page reports ablations across eight experiment families, covering output heads, patch and tubelet sizes, pixel versus VAE-space training, model scale and GPU count, and conditioning for image-to-video and video editing. On understanding benchmarks, the released checkpoint is described as comparable to unified models and VLMs such as BAGEL and TUNA. The authors note that differences in training data mean these results do not show one architecture is better.

Known limits: the released checkpoint uses a linear pixel output head, which can produce faint grid-aligned patch artifacts in smooth regions such as sky. These artifacts get worse at high classifier-free guidance (around CFG 6). Convolutional heads reduce them but need further training and are not in the released model. A YouTube reviewer described output quality as below frontier models, consistent with its research-preview status, and said the code and model are released mostly under Apache 2.0.