XNews

Luma Introduces MARBLE Sequence Parallelism for Packed Multimodal Masks

Luma Introduces MARBLE Sequence Parallelism for Packed Multimodal Masks

Luma introduced MARBLE, a mask-aware sequence parallelism method for packed multimodal sequences that selects ring attention, sparse all-to-all or Ulysses per layer. MARBLE matches single-GPU numerics and outperforms a FlashMask plus Ulysses hybrid in 47 of 48 cases on up to 128 H100 GPUs.

Event version 1.

Key insights

Per-layer strategy selection

MARBLE chooses ring attention, a sparse all-to-all or Ulysses for each layer based on mask patterns. The approach targets packed multimodal sequences where text is causal and images or video are usually bidirectional.

Exact numerics

MARBLE keeps the softmax state in fp32 across ring steps to avoid rounding errors. Its outputs and gradients match single-GPU results.

Balanced ring attention core

MARBLE uses a block-sparse planning strategy with per-token metadata and processing-time round-robin to balance work evenly across ranks. It also includes a chunked ring attention variant for long generations.

Benchmark results

MARBLE beats a FlashMask plus Ulysses hybrid in 47 of 48 cases. Testing ran on up to 128 H100 GPUs.

Source posts