Publish fake stats, get cited in LLMs

Original Article ↗ Hacker News Discussion ↗

A blog experiment describing how a $50 website of fabricated statistics was ignored by Google but ingested and cited by large language models has rekindled worries about AI systems laundering misinformation into seemingly authoritative answers. Commenters debate whether LLMs can ever reliably fact-check web content, given incentives for spam, SEO gaming, and “data poisoning,” and how much this amplifies long‑standing problems of online trust. Proposed responses range from stricter curation of training data and reliance on vetted “official” sources to broader societal skepticism about machine‑generated information.

Overview

  • Thread reacts to an experiment where a site with entirely fabricated statistics was cheaply created and then cited by LLMs, academic work, and bank research.
  • Core concerns: data poisoning, trust in AI outputs, responsibility for misinformation, and whether this is a new problem or a sped‑up version of old ones.

LLMs and Fake Data Vulnerability

  • Many argue LLMs “don’t know or care” if data is fake; they pattern‑match and confidently present results without an epistemic layer.
  • Fact‑checking via prompts is seen as unreliable: models can hallucinate having checked sources, over‑trust recent or popular content, and be more easily duped than humans while sounding authoritative.
  • Some note LLMs often pull from top search results, including SEO slop or LLM‑generated pages, creating feedback loops (“eating their own poo”).

Human Analogy and Old Problems

  • Several comments point out this predates LLMs: SEO spam, Wikipedia citogenesis, astroturfing, trap streets, fake reviews, and low‑quality outlets laundering “facts.”
  • Others stress that when humans browse, they see site context and can judge biases; LLMs strip that context and repackage everything as neutral expertise.

Ethics of Publishing Fake Stats

  • Strong disagreement over the experiment’s morality:
    • Critics call it lying, hostile to humanity, and akin to sabotaging the information ecosystem just to “prove” a known vulnerability.
    • Defenders say at least one public, well‑documented case is useful to expose the scale of the problem.
    • A minority goes further, openly endorsing “poisoning the web” to degrade AI systems, even accepting collateral damage.

Potential Mitigations and Limits

  • Proposed fixes:
    • Curated, high‑trust training corpora and avoiding open‑web search.
    • Direct connections to official data sources (e.g., statistics agencies, software docs).
    • Better source‑quality scoring, PageRank‑like weighting, provenance and RAG with traceable origins.
    • Detection of LLM‑generated text, though many see this as an adversarial arms race.

Broader Social Effects and Incentives

  • Some see this as part of a “knowledge lemon market”: incentives favor cheating and misinformation.
  • Views on regulation split: from calls for stronger fraud/consumer‑protection enforcement to fears of governments arbitrating “truth.”
  • Several predict declining trust in online information and a renewed emphasis on skeptical reading, trusted institutions, or even more face‑to‑face interaction.