The Curse of Recursion: Training on generated data makes models forget (2023)
Training AI models on their own generated outputs can lead to “model collapse,” where performance degrades as errors and stylistic quirks are amplified over generations, much like repeatedly recompressing a JPEG. Commenters contrast this with successes like AlphaZero, arguing that language lacks clear objective signals and that synthetic text often behaves like lossy, low-entropy data unless carefully filtered or mixed with substantial real-world input. The thread also highlights that access to fresh, human-produced data may become a key competitive advantage and notes emerging research suggesting that retaining original data alongside synthetic samples can mitigate collapse.
Nature of Synthetic vs Real Data
- Many argue the core issue isn’t “synthetic” per se but low‑quality, lossy, self‑generated data.
- Fiction (e.g., novels) is defended as real data about language and culture, not “simulated” worlds.
- Others insist that correctly generated synthetic data can be useful, e.g., game self‑play, simulations, or CGI images, but only if grounded in real distributions.
Information Loss, Entropy, and Feedback Loops
- Several comments frame recursive training as repeated application of a lossy, non‑invertible function, inevitably degrading information.
- References to data processing inequality and entropy: you can’t “cheat” physics; repeatedly compressing/compressing‑like transforms causes drift toward noise or blandness.
- Counterpoint: lossy transforms can sometimes help (denoising, structure extraction), so “loss = worse” isn’t universally true.
Model Collapse and Mitigations
- Broad agreement: purely replacing real data with model outputs leads to “model collapse” and degradation.
- Follow‑up research is cited: if synthetic generations are accumulated alongside original real data, collapse is avoided and performance is bounded.
- Some suggest quality filters, human feedback, and metadata (scores, links, timelines) can help exclude junk outputs from future training.
Human Learning Analogies and Limits
- Debate over whether humans are “immune”: most say no—science progresses by adding new experiments (new data) and discarding errors.
- Comparison: repeated human teaching works because it’s grounded in a stable external reality; current LLMs lack continuous real‑world interaction.
Use Cases for Synthetic Data
- Synthetic data can work well in narrow, supervised tasks (e.g., balancing labels in classification, distillation from larger to smaller models).
- Concern that over‑reliance on upscaled or hallucinated data (e.g., “enhanced” license plates) introduces false information and serious downstream risk.
Data Monopolies and Detection
- Consensus that fresh, genuine human interaction data becomes more valuable as the web fills with AI text.
- Large platforms with deep tracking and engagement signals are seen as having a major advantage.
- Some call for dedicated detectors and provenance systems, but others expect an ongoing arms race with no perfect solution.