Anyone got a contact at OpenAI. They have a spider problem

Original Article ↗ Hacker News Discussion ↗

A long‑running “honeypot” content farm with billions of near-empty subdomains has revealed that OpenAI’s GPTBot is aggressively crawling it despite `robots.txt` directives, raising questions about how responsibly large AI companies manage web scraping. Commenters explore technical angles (IP blocking, tarpit-style responses, and how subdomain structures can amplify crawler load), as well as legal and ethical issues around robots.txt, implied licenses, and whether scraping public pages is fair game. Many tie this to a broader concern that indiscriminate crawling—especially as more of the web becomes AI‑generated—could both waste resources and degrade future training data quality.

Honeypot content farm and OpenAI’s spider

  • The referenced site is a deliberate “honeypot” / “bot motel” with ~6.8B single-page subdomains, designed to:
    • Waste time and resources of crawlers.
    • Reveal new or poorly behaved spiders via logs.
  • It includes affiliate and book links, partly as low‑quality signals and possibly minor monetization.
  • OpenAI’s GPTBot reportedly made millions of requests, including ~1.8M for robots.txt, suggesting substantial crawl resources may be wasted on this trap.

Robots.txt compliance and crawler behavior

  • Core complaint: GPTBot appears to ignore robots.txt rules that disallow it, continuing to fetch pages and discover new subdomains.
  • Others note that:
    • Robots.txt is per-domain; with every page on a different subdomain, a compliant bot must refetch robots.txt for each.
    • The site owner later commented out the GPTBot disallow lines, so logs and current config differ over time.
  • Possible explanations discussed:
    • Bot discovered many subdomains before disallow was added.
    • Bot reads disallowed pages once (for headers/links) before honoring robots.
    • There may be external pages linking to millions of subdomains.
  • Some argue this “scan-first-respect-after” behavior is indistinguishable from not respecting robots at all.

Scraping legality and norms

  • Strong consensus: scraping publicly accessible (no-login) pages is broadly legal in the U.S., citing court precedent distinguishing public vs. auth‑walled content.
  • Debate over robots.txt:
    • One side: purely advisory, no legal force.
    • Other side: an implied license analogous to repo licenses; ignoring it could be a civil violation.
  • Ethical split between “it’s legal, scrape away” and “courtesy and self‑regulation matter, or laws will follow.”

Tarpits, DoS, and model poisoning

  • The farm is likened to a tarpit: infinite or near‑infinite pages can:
    • Amplify a crawl queue until it’s dominated by the honeypot.
    • Waste bandwidth/compute of large crawlers.
  • Ideas floated:
    • Slow‑loris style robots.txt responses or huge robots.txt files to DoS crawlers.
    • Using infinite auto‑generated content (possibly by LLMs themselves) to poison training data or bias models.
  • Counterpoint: large players can rate‑limit, deduplicate, and filter low‑value or synthetic content.

Long‑term AI training issues

  • Concern that as AI‑generated content dominates the web, future models will train on polluted, self‑referential data (“ouroboros”), degrading quality.
  • Others argue:
    • High‑quality curated or synthetic datasets (e.g., from textbooks) and RAG over “known good” knowledge will mitigate this.
    • Multimodal data and non‑web sources will increasingly matter.
  • Several discuss “hallucinations” as mostly about data quality, incentives, and lack of robust “I don’t know” behavior, not just model architecture.