XNews

SkyRL Enables 1M Token Context Training on GLM-5.3-Flash, 64x Megatron Baseline

SkyRL Enables 1M Token Context Training on GLM-5.3-Flash, 64x Megatron Baseline

SkyRL, in collaboration with Lila Sciences, enabled 1M token context training on GLM-5.3-Flash, achieving a 64x increase over the out-of-the-box Megatron baseline. The work used four optimization areas: DeepSeek Sparse Attention kernel fixes, activation recompute and CPU offloading, context parallelism, and removal of unnecessary FP32 upcasts. Peak memory reached approximately 144.5 GiB on B200 GPUs.

Event version 1.

Key insights

Four optimization areas

The work involved DeepSeek Sparse Attention optimizations, activation recompute and offloading, context parallelism, and avoiding unnecessary precision upcasts. These optimizations enabled training at 1M tokens per sequence.

DSA kernel fallback fixed

GLM's DSA layers were silently falling back to dense attention because the fused sparse kernel only recognized DeepSeek-V3.2 tensor shapes. Padding tensors to the expected shape and using TileLang reduced 16k memory from about 114 GiB to about 56 GiB per GPU.

Indexer chunking and sharding

Chunking the indexer by query reduced peak memory at 32k from about 64 GiB to about 2.3 GiB. Sharding the indexer across tensor-parallel ranks cut a 256k step on 64 B200 GPUs from about 218 seconds to about 102 seconds.

Real-data memory and correctness

On real book data, 512k peak memory was about 122 GiB and 1M with CP=4 was about 109 GiB. Trainer and vLLM logprob mean absolute difference was about 0.04–0.07 across 128k–1M contexts.

Long-context learning demonstrated

Fine-tuning on human genome hg38 with chr1–2 for training and chr22 held out showed held-out loss dropping about 4.8% over 80 steps at 512k. Position-bucketed results showed greater improvement on later tokens.

Source posts