OpenAI's bot crushed this seven-person company's web site 'like a DDoS attack'
Aggressive scraping by AI companies is overwhelming some small websites, driving up bandwidth costs and even causing outages, as illustrated by a media-heavy e‑commerce site that likened OpenAI’s crawler traffic to a DDoS attack. Commenters debate where responsibility lies: some argue site owners must use tools like robots.txt, rate limiting, and CDNs, while others contend that large AI firms have an ethical duty to crawl politely, honor blocking signals, and avoid imposing hidden infrastructure costs. The growing tension is pushing more content behind logins or bot blocks, which could reduce the amount of publicly accessible information on the web.
Legal and liability questions
- Commenters discuss whether excess hosting costs from scraping are legally recoverable.
- Cited case law suggests scraping public data is generally not a criminal CFAA issue; disputes are mostly civil.
- One small site reports successfully getting a few thousand dollars from an AI company for bandwidth overuse.
- Several note robots.txt has no legal force; lawsuits would likely rest on general claims of harm, not robots violations.
- There is disagreement on whether large-scale, non-consensual scraping could realistically succeed in court; some say “no precedent,” others point to past crawling lawsuits (unclear which).
Who is responsible for the overload?
- One camp: if you run a public site without auth, rate limits, caching, or robots.txt, you should expect heavy crawling and design for it.
- Opposing camp: small businesses can’t all be infra experts; bots that knock sites offline are behaving unreasonably, regardless of site quality.
- Analogies (e.g., emptying a free library, filling a shop with non-buyers) are used to argue that “not illegal” ≠ ethical.
Crawler behavior and engineering quality
- Many describe AI crawlers as poorly engineered: over-aggressive, ignoring 429 / Retry-After, re-crawling unchanged content, and sometimes spoofing user agents.
- Some note that classic search bots historically provided support channels and honored robots more reliably.
- Others argue the article’s “DDoS-like” framing is unproven because no request rates or timestamps are shown; Cloudflare IPs in logs further muddy attribution.
Mitigations and countermeasures
- Suggested defenses: robots.txt (with explicit AI blocks), Cloudflare protection and AI-bot blocking, fail2ban, HTTP 429 with subnet-level throttling, ASN or country blocks, IPv6-only access, and .htaccess rules.
- Some propose “data poisoning” defenses: serving gibberish, recursive content, or compressible text to abusive bots; others argue such gibberish is easy to filter in curation.
Broader implications for the web and AI
- Concern that aggressive AI scraping will push more content behind logins/paywalls, reducing open information.
- Some see AI agents as reviving “personal webcrawlers” and automating interactions with sites that don’t offer APIs.
- Others worry this simply recreates a centralized, Google-like gatekeeper, now controlled by AI companies.