OpenAI's Jack Rae frames LLM training as lossless compression of data
Jack Rae (OpenAI) proposed that training LLMs is equivalent to lossless compression: better models compress training corpora more tightly, explaining generalization and why larger models can be 'simpler' by producing smaller compressed-data size. Compression provides an evaluation metric robust to test contamination. The perspective links successive ML paradigm shifts to improved compression and frames AGI progress as minimizing description length of useful percepts.