A year of fighting scrapers on my 1.5 million-page website

Original Article ↗ Hacker News Discussion ↗

Web publishers report that the vast majority of their traffic now comes from bots—especially AI crawlers and poorly written scrapers—which can saturate bandwidth, distort analytics, and inflate hosting bills while sending little or no human traffic in return. Commenters describe a range of defenses, from Cloudflare, geo‑blocking and proof‑of‑work challenges to static hosting and aggressive firewall rules, but note these often harm legitimate users and further centralize the web. Underneath is a broader conflict between preserving an open, machine-readable internet and protecting small sites from being effectively DDoS’d and data-mined without compensation.

Scale and Nature of Bot Traffic

  • Many operators report 95–99.9% of traffic as bots, especially AI crawlers from major US and Chinese companies and residential proxy networks.
  • Bots often behave poorly: re-scraping unchanged pages, walking infinite calendars, hammering expensive endpoints, probing for secrets (.env, keys, Dockerfiles).
  • Some suspect repeated refetching is due to badly written agents and “scrape everything” VC-fueled attitudes, not user-driven queries.

Impact on Small Sites and Hosting

  • Several report formerly cheap hobby or niche sites becoming unsustainable as bandwidth and database load jump by orders of magnitude with little or no revenue.
  • Others counter that a well-built static or efficiently cached dynamic site on a $4–5 VPS can easily handle current bot levels; problems are often slow stacks or expensive DB queries.

Defenses and Their Tradeoffs

  • Common tactics:
    • CDN/WAF in front (Cloudflare et al.), including new “block AI bots” and pay‑per‑crawl features.
    • Geo-blocking or CAPTCHAs for certain countries/ASNs and cloud providers.
    • Proof‑of‑work / JavaScript challenges (e.g., Anubis), cookie checks, HTTP header heuristics, TLS fingerprinting.
    • Moving content behind logins or paywalls; pre-generating static pages.
  • Proof‑of‑work systems spark debate: critics show they’re trivial to bypass with native/CUDA solvers; defenders say they still stop the vast majority of unsophisticated scrapers and are less annoying than CAPTCHAs.
  • False positives are real: privacy browsers, JS-disabled users, VPN users, and older systems get blocked or slowed, sometimes permanently.

Centralization, Open Web, and Tooling

  • Heavy reliance on Cloudflare and similar is viewed by some as necessary protection; others see it as turning the web into walled gardens controlled by a few gatekeepers.
  • Some argue the “social contract” of the web includes bots and alternative user agents; others say that contract never covered LLM training at scale.
  • Many note AI crawlers send negligible referral traffic while extracting high value, prompting calls for payment or stronger legal/technical restrictions.
  • Robots.txt “content signals” and EU copyright reservations are mentioned, but their real enforceability remains unclear.
  • A growing pattern: people shutting down or de‑indexing hobby sites, or tolerating only human traffic, even if that means accepting some user inconvenience.