OpenAI Researcher Frame: Training LLMs Is Essentially Lossless Compression
Jack Rae framed LLM training as lossless compression: finding minimal executable descriptions reduces model perplexity and improves generalization. The post cites Llama 65B compressing 5.6TB of text to ~400GB compressed-data equivalent, explaining why larger models can be 'simpler' in description length and why compression resists test-set contamination as a cheating strategy.