OpenAI Researcher Jack Rae: Training LLMs Is Essentially Lossless Compression
Jack Rae framed large model training as lossy/lossless compression: a model that predicts tokens effectively compresses datasets—e.g., Llama-65B can compress 5.6TB of text down to ~400GB compressed representation—so better compression equals better generalization. This view links Solomonoff-style minimum description length to scaling, explains emergent capabilities as compression thresholds and highlights limitations for high-dimension data and need for active exploration.