Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

Original Article ↗ Hacker News Discussion ↗

Mass, automated vulnerability scans against web servers are increasingly disguising themselves as legitimate AI crawlers like Claude or Googlebot, with some operators apparently leveraging cloud providers and certificate transparency logs to discover and probe new sites quickly. Commenters note that indiscriminate blocking of VPS IP ranges, ASNs, or entire countries can cut a lot of hostile traffic but also excludes real users, pushing sites toward more nuanced defenses such as honeypots, behavioral signals, and bot authentication schemes like Google’s Web Bot Auth. The exchange underscores that background scanning is a permanent “water is wet” condition of the internet, but the scale, sophistication, and AI-related targeting of these campaigns are clearly ramping up.

Nature of the scans

  • Many see a recent statistically significant surge in HTTP(S) requests spoofing AI bot user-agents (Claude, Googlebot, etc.), often failing IP verification or new “Web Bot Auth” schemes.
  • Paths requested frequently target AI coding tools and common web vulns (e.g., WordPress, .env, generic PHP endpoints).
  • Some argue this is just the same old mass vuln scanning with new disguises; others see a distinct new campaign pattern.

Traffic sources and patterns

  • Reported origins include Google Cloud (specific ASNs), Microsoft and Google IPs, Chinese and Russian networks, and various VPNs, VPSes, and compromised residential/IoT devices.
  • Traffic often ramps and falls in lockstep across thousands of IPs, suggesting centralized control.
  • Certificate Transparency logs and new TLS cert issuance are cited as triggers for bots to find fresh domains quickly.

Bot identification and verification

  • Consensus that user-agent strings are unreliable; verification should use:
    • Provider-published IP ranges or reverse DNS.
    • New Web Bot Auth mechanisms where supported.
  • Some see fake Googlebot/AI crawlers as the top category in logs.

Mitigation strategies

  • IP/ASN-based:
    • Blocking VPS/datacenter ASNs; using free or paid IP intelligence (from cheap to very expensive enterprise tiers).
    • Some block entire ASNs like large foreign providers; others caution about false positives.
    • Country-level blocking at routers/firewalls is common, sometimes combined with ASN blacklists.
  • Behavioral:
    • Honeypots that collect scan traffic and publish blocklists and AS/ISP leaderboards.
    • Dynamic systems that gate “risky” networks behind CAPTCHAs/Turnstile based on indicators (e.g., many PHP probes, many empty accounts per /24).
    • Rate limiting, fail2ban, and WAFs; critics say fail2ban is overwhelmed by millions of IPs and mostly cleans logs.
  • Architectural:
    • Moving from WordPress/CMS to static hosting (S3/Cloudflare Pages) to shrink attack surface.
    • Using HTTP/2-only or header fingerprinting to filter some bots.
    • Relying on CGNAT or WireGuard-style VPNs to reduce exposure.

Tradeoffs and collateral damage

  • Blocking VPS/datacenters/VPNs inevitably locks out some real users; anecdotes suggest the absolute number is small but non-zero.
  • Residential proxies and CGNAT mean a single blocked IP or town-sized NAT can affect many innocents.
  • Some prefer passive hardening and reacting only to worst offenders rather than aggressive bot fighting that might block legitimate crawlers or users.

Legality, norms, and significance

  • Mass unauthorised scanning is widely viewed as illegal or “legally dubious,” but enforcement is seen as minimal.
  • Disagreement over novelty: some call it business-as-usual since early worms; others point to rising volume and AI‑targeted paths as evidence of escalation.
  • Several see this as part of a broader shift toward more authenticated, less open internet access (e.g., KYC-like models, bot auth standards).