Can gzip be a language model?
Can traditional compression tools like gzip act as crude language models by choosing continuations that compress best? Commenters explore the deep link between compression and prediction, citing experiments where gzip and other codecs are used for text generation, classification, and even image or digit recognition, while noting that search over possible continuations is intractable and naive objectives quickly degenerate into repetitive output. The thread contrasts these approaches with modern neural language models, referencing work like the Hutter Prize and ts_zip to argue that while compression quality is a useful proxy for some aspects of “intelligence,” LLMs capture far richer structure at vastly higher computational cost.
Gzip as a Language Model
- Several commenters note the deep theoretical link: next‑token prediction and compression are closely related (cross‑entropy, MDL).
- The experiment of using gzip to score continuations illustrates this connection, but most see it as a toy model rather than a rival to neural LMs.
Limits of the Gzip LM Approach
- A core criticism: the search space of possible continuations is astronomically large, so beam search over gzip scores only finds a lower bound, not true optima.
- One commenter shows that simply copying a matching span from the context (“rfind” strategy) yields much better compression scores than the beam search, but is useless as a generative model.
- Others point out that to avoid trivial solutions (e.g., repeated characters, large backreferences), the author had to restrict the context window and tune the method until outputs “looked interesting,” weakening the premise.
Comparisons with Other Compression Algorithms
- Experiments swapping gzip for bzip2, zstd, lzma, and lz4 often produce degenerate outputs (alternating characters, mostly whitespace, or only newlines), reinforcing how codec details shape “generated” text.
- Separate subthread: discussion of zstd vs gzip vs LZMA2 for general-purpose compression; zstd praised for speed/ratio, LZMA for archival ratio.
Compression, Intelligence, and LLMs
- Multiple references to the Hutter Prize and a known project using LLMs for text compression, which can beat strong traditional compressors (ignoring speed and model size constraints).
- Some argue compression ability is a plausible proxy for one dimension of “intelligence,” but misses creativity and other aspects.
- Others stress that LLMs are more like lossy codecs (JPEG/MP3) than like gzip; hallucinations are likened to lossy artefacts.
Classification and Other Uses of Compression
- Commenters describe using gzip/zstd/LZO-like methods for topic classification, spam detection, and language detection via “which reference corpus compresses best,” tied to normalized compression distance.
Meta and Philosophical Points
- Debate over whether the article itself was AI-assisted and whether that undermines its value.
- Several point to educational resources (videos, textbooks, prior papers) on the compression–language modeling connection.