How to Block Some of the Bots

Original Article ↗ Hacker News Discussion ↗

Aggressive techniques for blocking web crawlers and AI scrapers—such as banning data-center IP ranges, VPNs, curl/wget, and even entire continents—are drawing mixed reactions from site operators and users. Some argue that with bandwidth costs, abusive bot swarms, and DDoS-like traffic, heavy-handed blocking is a practical necessity and a fun technical challenge, especially compared to outsourcing everything to Cloudflare. Others counter that these tactics break access for legitimate users, entrench major browser and infrastructure monopolies, and risk making independent sites invisible to people who increasingly rely on LLMs and alternative clients to navigate the web.

Access problems & collateral damage

  • Many commenters cannot reach the article at all (TLS errors, HTTP 410, blank pages), including on mainstream browsers and iOS Safari.
  • Blocking of VPNs, data centers, some mobile networks, and even “two entire continents” is reported.
  • Some see this as over‑zealous and hostile to normal users; others accept it as a deliberate “demo” of extreme blocking.
  • An archive mirror is appreciated as a workaround, but some still can’t access even that in certain setups.

IP-, UA-, and header-based blocking

  • The article’s techniques include datacenter IP blocking, HTTP/1.1 blocking, and header checks (e.g., Sec-Fetch-Mode, X-Forwarded-For).
  • Critics warn that:
    • Datacenter/VPN classification is unreliable and can hit legitimate users, including schools, libraries, and Firefox’s built-in VPN.
    • IP ranges and ASNs get reassigned, so blanket bans can later affect residential ISPs.
    • Blocking on referer patterns (e.g., from Google) can also affect real search traffic.
  • Supporters argue that site owners are entitled to aggressively protect limited resources and time.

Curl, wget, and alternative clients

  • Blocking curl/wget is widely disliked, especially for downloading artifacts from servers.
  • The author’s stance is to add “friction,” expecting only unsophisticated bots to be stopped; critics respond that users will just spoof a browser UA.
  • Some argue that blocking “non-approved” user agents entrenches dominant browsers and harms the open web.

Bots, costs, and motivations

  • Several operators describe real harm from bots:
    • Overeager or stupid crawlers creating redirect loops, hammering dynamic endpoints, and DDoS‑like traffic (tens of thousands of req/minute, millions of URLs/day).
    • Shared hosting suspension, bandwidth caps (e.g., 2 TB/month mostly eaten by bots), CPU exhaustion on dynamic services like git forges or VCS web UIs.
  • Others say typical blogs/static sites can easily handle bot load and suspect some blocking is “for the thrill of the game.”

Residential proxies, AI crawlers, and the future web

  • Datacenter blocking pushes serious scrapers toward residential and “premium” proxy networks; some see these as effectively “voluntary malware.”
  • One commenter claims top-tier AI data vendors already bypass such defenses completely via residential IPs and browser fingerprint replay.
  • There’s debate on whether it’s desirable to block LLM/AI access entirely, given many people increasingly experience the web via such tools.