XNews

Multimodal world models and image text rendering

Multimodal world models and image text rendering

New multimodal world models aim to predict next-world states and improve in-image text and instruction following. - Emu3.5 (Beijing Academy of Artificial Intelligence) is a unified multi-modal world model that tokenizes image/text/video and trains on next-state prediction (NSP) to predict the 'next world state' across modalities. - Emu3.5 treats images, frames, and text uniformly as tokens and learns to predict next tokens representing subsequent visual or textual states. - Nano Banana Pro and GPT-Image-1 show significant advances in text rendering and instruction-following in images by using native multimodal transformer token prediction rather than pixel-only diffusion. - Native token-based multimodal models generate in-image text with much lower spelling error rates because they treat characters as tokens and 'write' them coherently across modalities. - These approaches enable better spatial reasoning, causal prediction across frames, and higher-fidelity instruction adherence in complex visual compositions.

Key insights

Emu3.5 unifies modalities under next-state prediction

Emu3.5 is a multimodal world model that converts images, text, and video into a shared token representation and trains the model to predict the next world state token (NSP). By treating all modalities uniformly, the model learns cross-modal dynamics and temporal prediction—predicting subsequent frames, continuing captions, or the next event—aiming to improve spatial reasoning and temporal continuity across vision and language tasks.

Native multimodal token models improve in-image text generation

Nano Banana Pro and GPT-Image-1 generate image text and follow layout instructions more accurately than diffusion-based image generators. Because token-based multimodal transformers predict text tokens directly in a shared semantic space, they render characters with far fewer spelling errors and better alignment to instructions compared with pixel-level denoising diffusion methods that treat characters as unconstrained pixels.

World-model training objective focuses on causal and temporal prediction

The NSP objective used by Emu3.5 reframes multimodal generation as a next-state prediction problem: the model learns to forecast the next visual or textual token sequence for a given world state, enabling causal reasoning across time and modalities rather than isolated generation per modality.

Multimodal tokenization supports unified reasoning and higher fidelity outputs

Converting images, video frames, and text into a shared token vocabulary allows transformers to apply the same predictive machinery to all modalities, improving cross-modal coherence and enabling tasks like multi-frame prediction, instruction following in visual layouts, and consistent rendering of textual content inside images.

Source posts