Compression is prediction

Original Article ↗ Hacker News Discussion ↗

Compression and prediction are increasingly being treated as two sides of the same coin, with many pointing out that modern AI models—especially large language models—can be seen as powerful, lossy compressors of their training data. Commenters connect this idea to Shannon’s information theory, Kolmogorov complexity, the Hutter Prize, and classic compression techniques, noting that better prediction implies better compression and that this may explain how abstract “world models” and seemingly novel ideas emerge from training. Others push back on over-simplified slogans like “compression is intelligence,” arguing that generalization, creativity, and real-world knowledge require more than optimal coding of past data, and emphasizing the importance of experimental validation and historical context for these concepts.

Established idea: compression, prediction, information theory

  • Many point out this is standard information‑theory: good compression requires good probabilistic prediction of the next symbol.
  • Statistical compressors model distributions and encode likely events in fewer bits; this is directly analogous to next‑token prediction in language models.
  • Some stress that this connection has been known for decades, with prior work on inductive inference, Kolmogorov complexity, and compression‑based similarity measures.
  • Others emphasize that “compression is prediction” is an oversimplification: compression uses prediction, but is not identical to it in all settings.

LLMs, world models, and “new ideas”

  • Several comments use the compression view to counter “LLMs are just parrots” claims: training can be seen as finding a compact model of the data‑generating process, not just memorizing curves.
  • Analogy: fitting a compact function to planetary motion lets you predict unseen orbits; similarly, a language model approximates the processes that generated human text and can generalize beyond the training set.
  • Some argue “new ideas” in humans are also recombinations/deductions from prior data; in that sense, advanced models could do the same.
  • Others insist prediction alone cannot produce new knowledge about the physical world without experiments.

Limits, generalization, and distribution shift

  • Critics note a key caveat: prediction≈compression only holds cleanly when training and test distributions match.
  • Overfitting for maximal compression on a fixed dataset can hurt performance on future or shifted data; this is likened to classic generalization/overfitting trade‑offs.
  • Some respond that any compressor that better models the true source will eventually compress future data better too, but agree this is distribution‑dependent.

Compressors as generators (and vice versa)

  • Multiple comments explain that any probabilistic compressor can be turned into an autoregressive generator: try each possible next token, see which co‑compresses best, and sample accordingly.
  • Experiments with neural compressors and LLM‑based compressors show strong results on both in‑distribution and held‑out text, often beating traditional tools in bits per byte.
  • However, traditional compressors still make very poor text generators in practice, illustrating that “being a compressor” is necessary but not sufficient for high‑quality prediction.

Understanding, abstraction, and lossy vs lossless

  • There is an extended side‑discussion on whether “understanding is lossy compression”:
    • One camp: abstraction/generalization is exactly selective information loss, i.e., deciding which details don’t matter.
    • Another camp: you can compress syntax losslessly with zero semantic understanding; high‑quality lossy compression may correlate more with genuine understanding.
  • Teaching vs understanding vs application are framed as different phases of compression/decompression in humans.

Intelligence, consciousness, and scope

  • Some see tight links among compression, intelligence, and even aesthetics (beauty as efficient description, physics as ultimate compression).
  • Others warn against overextending: LLMs being good compressors does not by itself imply consciousness; by that logic, ordinary file compressors would be candidates too.
  • Still, many expect that increasingly powerful compression/prediction systems will require increasingly rich world models.

Critiques of the article and pedagogy

  • Several commenters like the visuals and think the piece is an accessible exposition for non‑experts.
  • Others criticize the presentation as if the ideas were newly “discovered,” with too little historical context or citation of older work.
  • There is debate over how much background and attribution a popular blog post should include versus keeping explanations lightweight and engaging.