I posted to HN, and then my website was flooded with AI bot traffic

Original Article ↗ Hacker News Discussion ↗

Posting a link to Hacker News now often triggers a surge of automated traffic, from aggressive AI-driven scrapers to generic vulnerability scanners, that can overwhelm small sites and distort analytics or even generate large cloud bills. Commenters trade tactics—blocking cloud provider ASNs, CAPTCHAs like Cloudflare Turnstile, tarpits, proof-of-work, basic auth, geo-allowlists, and static hosting—while noting that sophisticated residential botnets and headless browsers make permanent technical fixes elusive. Many foresee this trend pushing more content behind logins or walled gardens, raising concerns about the future openness and usability of the web.

HN exposure and bot swarms

  • Multiple commenters report that merely posting on HN (even without reaching front page) triggers large spikes of automated traffic.
  • Traffic often far exceeds legitimate user visits and continues for months after a single post.
  • Some note that request volume is within what modern servers can technically handle, but still causes slowdowns, outages, or big bills when it hits dynamic or poorly cached paths.

Nature of current bot ecosystem

  • Mix of:
    • Classic “internet background radiation”: vulnerability scans, WordPress/PHP probes, wp-content/phpinfo.php, etc.
    • Cloud-hosted scrapers from AWS, GCP, Azure, Alibaba, Oracle, Meta, Apple, and others.
    • AI/LLM-related crawlers and “AI newsletter”/change-monitoring services polling far too frequently.
    • Residential proxy botnets and consumer VPNs, making IP-based blocking and fail2ban far less effective.
  • Some bots ignore robots.txt or nofollow, and keep re-hitting long-dead URLs for years.

Mitigation strategies and trade-offs

  • Heavy blocking: by ASN (clouds), GeoIP allow-lists, AbuseIPDB blocklists, blackhole/“trap” URLs, or blackhole routing at the network edge.
  • Challenges and CAPTCHAs: Cloudflare Turnstile, Anubis-style JS work, proof-of-work, custom L7 proxies and tarpits that deliberately slow responses.
  • Simpler defenses: basic auth on everything but homepage, skill-testing questions, rate limiting, fail2ban/ipsets.
  • Many report that advanced bots now use real browsers/headless Chrome and defeat CAPTCHAs and Turnstile.
  • Several conclude the arms race is unwinnable and instead focus on performance optimizations, static content, CDNs, and restricting dynamic features to authenticated users.

Impact on hosting, costs, and analytics

  • Reports of multi-thousand-dollar cloud bills and high egress costs from aggressive scraping.
  • Some self-hosters block large clouds despite collateral damage to legitimate users behind work VPNs.
  • Analytics and logs are described as “useless” or heavily polluted; some use LLMs to find patterns and classify bad traffic.

Debate on LLM/search bots and the future web

  • Some see value in LLM referrers, claiming ChatGPT is becoming a top traffic source and yields highly engaged users.
  • Others argue these systems aim to keep users away from sites, providing little value to content owners.
  • Several predict increasing movement to walled gardens, strict identity requirements, or web-of-trust style networks as the open, anonymous web deteriorates under bot pressure.