Google is the only search engine that works on Reddit now, thanks to AI deal

Reddit’s robots.txt change and Google deal

• Reddit’s robots.txt now disallows all generic crawlers, while serving Google a different, more permissive version.
• Many see this as part of a broader strategy: close the API, block generic crawling, then sell data access (e.g., ~$60M Google AI deal).
• Some argue this is driven by financial pressure: Reddit is still loss‑making despite licensing revenue.

Impact on search and users

• Non‑Google engines (Bing, DDG, Mojeek, etc.) either lose fresh Reddit results or must buy access indirectly (e.g., via Google or licensing).
• Users who relied on site:reddit.com for high‑signal answers feel pushed back to Google.
• Others welcome the change, happy to see less Reddit in search results, claiming many threads are low‑quality or heavily moderated “hiveminds.”

Scraping, robots.txt, and legality

• Several comments note U.S. case law that public pages can be scraped regardless of robots.txt, though copyright and ToS still constrain use.
• Others point out technical blocks (datacenter IP bans, Cloudflare, anti‑bot features) make scraping costly even if legally permitted.
• Some suggest a future of “data laundering”: independent scrapers repackaging Reddit content for AI or search under fair‑use arguments.

Competition and antitrust concerns

• One side: blame lies almost entirely with Reddit; any search engine can pay too, so not anti‑competitive.
• Other side: in practice only giants can afford many such deals, raising barriers to entry and reinforcing Google’s search monopoly.
• Some think truly exclusive indexing deals could trigger antitrust scrutiny; whether this deal is exclusive is unclear.

Ethics of monetizing user-generated content and AI

• Strong disagreement over whether Reddit is ethically entitled to sell access to user posts.
• Some emphasize users hold copyright but have granted Reddit a broad license; others stress the moral problem of monetizing unpaid labor while restricting broader access.
• Many tie this to “enshittification” of the web: platforms closing off, chasing short‑term profit, and reacting to AI scraping by becoming walled gardens.

Alternatives and broader web trends

• Lemmy and federated “distributed Reddit” are mentioned, but network effects and moderation/spam burdens are seen as major obstacles.
• Some hope this fragmentation pushes people back to independent forums and hobbyist sites; others think LLM‑driven scraping and spam will only worsen.