Jack Rae (OpenAI) Frames Large Language Model Training as Lossless Compression, Citing Llama Numbers
The post presents Jack Rae's perspective that training large language models is equivalent to lossless compression: better compression = deeper understanding. It cites Solomonoff's theorem, the Chinese Room thought experiment, and Llama 65B compressing 5.6TB to ~400GB (14×), argues model weights need not be shipped, and frames compression as an un-gamable training objective with implications for scaling, architectures, tools, multimodal fusion and AGI limits.