Will AI systems perform poorly due to AI-generated material in training data?

Original Article ↗ Hacker News Discussion ↗

Watermarking and Detecting AI Output

  • Some assume large labs watermark LLM outputs (statistical patterns, etc.) to later filter them from training sets; others think watermarking was largely abandoned as unreliable.
  • Even if a vendor can track its own outputs, they cannot reliably filter outputs from competing models.
  • Observed “anti‑GPT-ism” phrasing in system prompts (e.g., suppressing stock moralistic phrases) is taken as evidence that newer models’ training data is already contaminated with AI text.

Quality and Role of Synthetic Data

  • Several commenters argue synthetic data is already central and beneficial:
    • Llama 3’s post-training reportedly uses almost entirely synthetic answers from Llama 2.
    • DeepSeek models and others are cited as heavily synthetic yet strong, contradicting simple “self‑training collapse” fears.
  • Synthetic data is framed as an extension of training: used for classification, enhancement, and generating infinite math/programming problems, not just copying the web.
  • Skeptics ask how synthetic data can exceed the quality of the original human data, especially in fuzzy domains without clear correctness checks.

Risk of Model Collapse and Error Accumulation

  • One camp: repeated training on AI outputs leads to compounding “drift error,” where small hallucination rates amplify across generations until output becomes mostly wrong.
  • The opposing camp: if selection/filters exist (human feedback, automated checks, tool use), retraining on model outputs can at worst preserve quality and often improves it.
  • Some compare self-play in games (e.g., chess/Go) as evidence that self-generated data plus clear rewards can produce superhuman systems; critics counter that most real-world tasks lack such clean reward signals.

Human Data, Feedback Signals, and Privacy

  • LLM chat logs are seen as a massive ongoing human data source, though many view the prompts/responses as low-quality or noisy.
  • Weak behavior signals (rephrasing, follow-up prompts, “thumbs up/down,” scrolling) are considered valuable at scale, but skeptics doubt they can match rich, organically written content.
  • There is concern about whether “opt-out from training” settings are genuine or dark patterns; enforcement ultimately depends on trust and legal penalties.

Reasoning vs Knowledge Base

  • Some argue future progress will come from improved “core reasoning” and tool use, while encyclopedic knowledge from raw web text becomes less central and more polluted.
  • Others question whether current chain-of-thought outputs demonstrate genuine reasoning or just plausible-looking text with unobserved jumps to the answer.

Broader Social and Cultural Feedback Loops

  • Worry that humans are already being “trained on LLM garbage” (homework, coding, medical study aids) and will produce more derivative, low-quality text, further polluting training data.
  • Counterpoint: human culture has always been self-referential; art and writing haven’t degraded just because humans learn from prior artifacts.
  • Some foresee models learning to detect AI slop as a robustness feature; others fear a cultural “enshittification” equilibrium where both humans and AIs converge on bland, GPT-like language.