Google is the only search engine that works on Reddit now, thanks to AI deal

Original Article ↗ Hacker News Discussion ↗

Reddit has updated its robots.txt to block most web crawlers, effectively making Google the only major search engine that can reliably index its content due to a paid data-licensing deal. Commenters see this as part of a broader trend of platforms enclosing previously open user-generated content to sell it for AI training and search, raising concerns about monopoly power in search, barriers for smaller engines, and the erosion of the open web. Many also question the ethics of Reddit monetizing users’ posts while restricting independent access, and worry this model will spread to other large sites.

Reddit’s robots.txt change and Google deal

  • Reddit’s robots.txt now disallows all generic crawlers, while serving Google a different, more permissive version.
  • Many see this as part of a broader strategy: close the API, block generic crawling, then sell data access (e.g., ~$60M Google AI deal).
  • Some argue this is driven by financial pressure: Reddit is still loss‑making despite licensing revenue.

Impact on search and users

  • Non‑Google engines (Bing, DDG, Mojeek, etc.) either lose fresh Reddit results or must buy access indirectly (e.g., via Google or licensing).
  • Users who relied on site:reddit.com for high‑signal answers feel pushed back to Google.
  • Others welcome the change, happy to see less Reddit in search results, claiming many threads are low‑quality or heavily moderated “hiveminds.”

Scraping, robots.txt, and legality

  • Several comments note U.S. case law that public pages can be scraped regardless of robots.txt, though copyright and ToS still constrain use.
  • Others point out technical blocks (datacenter IP bans, Cloudflare, anti‑bot features) make scraping costly even if legally permitted.
  • Some suggest a future of “data laundering”: independent scrapers repackaging Reddit content for AI or search under fair‑use arguments.

Competition and antitrust concerns

  • One side: blame lies almost entirely with Reddit; any search engine can pay too, so not anti‑competitive.
  • Other side: in practice only giants can afford many such deals, raising barriers to entry and reinforcing Google’s search monopoly.
  • Some think truly exclusive indexing deals could trigger antitrust scrutiny; whether this deal is exclusive is unclear.

Ethics of monetizing user-generated content and AI

  • Strong disagreement over whether Reddit is ethically entitled to sell access to user posts.
  • Some emphasize users hold copyright but have granted Reddit a broad license; others stress the moral problem of monetizing unpaid labor while restricting broader access.
  • Many tie this to “enshittification” of the web: platforms closing off, chasing short‑term profit, and reacting to AI scraping by becoming walled gardens.

Alternatives and broader web trends

  • Lemmy and federated “distributed Reddit” are mentioned, but network effects and moderation/spam burdens are seen as major obstacles.
  • Some hope this fragmentation pushes people back to independent forums and hobbyist sites; others think LLM‑driven scraping and spam will only worsen.