Anyone got a contact at OpenAI. They have a spider problem

Honeypot content farm and OpenAI’s spider

• The referenced site is a deliberate “honeypot” / “bot motel” with ~6.8B single-page subdomains, designed to:
  • Waste time and resources of crawlers.
  • Reveal new or poorly behaved spiders via logs.
• It includes affiliate and book links, partly as low‑quality signals and possibly minor monetization.
• OpenAI’s GPTBot reportedly made millions of requests, including ~1.8M for robots.txt, suggesting substantial crawl resources may be wasted on this trap.

Robots.txt compliance and crawler behavior

• Core complaint: GPTBot appears to ignore robots.txt rules that disallow it, continuing to fetch pages and discover new subdomains.
• Others note that:
  • Robots.txt is per-domain; with every page on a different subdomain, a compliant bot must refetch robots.txt for each.
  • The site owner later commented out the GPTBot disallow lines, so logs and current config differ over time.
• Possible explanations discussed:
  • Bot discovered many subdomains before disallow was added.
  • Bot reads disallowed pages once (for headers/links) before honoring robots.
  • There may be external pages linking to millions of subdomains.
• Some argue this “scan-first-respect-after” behavior is indistinguishable from not respecting robots at all.

Scraping legality and norms

• Strong consensus: scraping publicly accessible (no-login) pages is broadly legal in the U.S., citing court precedent distinguishing public vs. auth‑walled content.
• Debate over robots.txt:
  • One side: purely advisory, no legal force.
  • Other side: an implied license analogous to repo licenses; ignoring it could be a civil violation.
• Ethical split between “it’s legal, scrape away” and “courtesy and self‑regulation matter, or laws will follow.”

Tarpits, DoS, and model poisoning

• The farm is likened to a tarpit: infinite or near‑infinite pages can:
  • Amplify a crawl queue until it’s dominated by the honeypot.
  • Waste bandwidth/compute of large crawlers.
• Ideas floated:
  • Slow‑loris style robots.txt responses or huge robots.txt files to DoS crawlers.
  • Using infinite auto‑generated content (possibly by LLMs themselves) to poison training data or bias models.
• Counterpoint: large players can rate‑limit, deduplicate, and filter low‑value or synthetic content.

Long‑term AI training issues

• Concern that as AI‑generated content dominates the web, future models will train on polluted, self‑referential data (“ouroboros”), degrading quality.
• Others argue:
  • High‑quality curated or synthetic datasets (e.g., from textbooks) and RAG over “known good” knowledge will mitigate this.
  • Multimodal data and non‑web sources will increasingly matter.
• Several discuss “hallucinations” as mostly about data quality, incentives, and lack of robust “I don’t know” behavior, not just model architecture.