World Models & Multimodal Prediction (Emu3.5, Runway advances)

Beijing Zhiyuan introduced Emu3.5, a multimodal world model that converts text, images and video into a shared token space and trains by predicting the next world state (NSP). Emu3.5 treats all modalities uniformly and learns temporal dynamics. Runway released Gen 4.5 and a GWM suite: ALF video editing with audio editing, GWM1 autoregressive frame prediction, GWM Worlds for immersive environment generation, GWM Avatars for audio‑driven high‑fidelity digital humans, and GWM Robotics for simulating both successful and failed interactions for robot training.
Key insights
Emu3.5 pursues unified multimodal next‑state prediction
Beijing Zhiyuan’s Emu3.5 treats images, video and text as a single tokenized modality and trains with next‑state prediction objectives. The model aims to learn physical and temporal regularities by predicting the next token whether it is a word or a video frame, thereby learning cross‑modal dynamics.
Runway advances video editing and world modeling
Runway announced Gen 4.5 with native audio generation/editing and ALF for consistent multi‑shot video edits. The GWM suite includes GWM1 for autoregressive frame prediction, GWM Worlds for interactive environment generation, GWM Avatars for realistic audio‑driven digital humans, and GWM Robotics for simulating both successful and failed robot interactions.
World models emphasize temporal and causal prediction over pixel reconstruction
Both Emu3.5 and Runway’s GWM prioritize predicting future states and dynamics in representational spaces, focusing on meaningful, controllable outcomes rather than pixel‑for‑pixel replication, enabling planning and intervention during generation.