Training LLMs as lossless compression: Jack Rae’s compression perspective on GPT
Presents the view that training large language models is essentially lossless compression: better compression implies deeper understanding and stronger generalization. Cites Solomonoff and compression-theoretic arguments, Llama 65B’s data compression ratios, and explains why scaling and architecture improvements that lower post-training data size equate to progress toward AGI.