XNews

Training LLMs Interpreted as Lossless Compression; Llama 65B Example Shows High Compression

Argument: training LLMs is effectively lossless compression of web-scale text. Using Solomonoff-style theory, a better compressor implies deeper understanding. Example: Llama 65B compresses 5.6TB of tokens to ~400GB of compressed data (14x), outperforming classical compressors, framing model scaling as improving data compression and generalization.

Source posts