The bot situation on the internet is worse than you could imagine

Original Article ↗ Hacker News Discussion ↗

Bots and large-scale scrapers are increasingly overwhelming websites with traffic that often looks indistinguishable from legitimate users, much of it routed through residential proxies and apparently driven by AI training and aggressive data collection. Commenters examine technical countermeasures such as proof‑of‑work systems like Anubis, IP/ASN blocking, and fingerprinting, noting that while some approaches can drastically cut bot traffic, they also risk crippling performance, blocking real users, and even resembling cryptomining. Underneath the technical details is a broader concern that the economic incentives around scraping and AI may erode the open web, advertising-supported sites, and user privacy, while heavy-handed defenses make the human browsing experience worse.

Anubis Proof-of-Work and User Experience

  • Many commenters couldn’t access the article due to Anubis PoW set at high difficulty: multi-minute or hour-long waits, phones/laptops running hot, high CPU and battery drain.
  • Some see it as effectively blocking the site rather than protecting it; suggestions that at this point you might as well take the site down.
  • Several suspect or joke that it looks like cryptomining or a “honey pot.”
  • Others point out experimental data: at low difficulty in a tar-pit, Anubis reduced hundreds of thousands of daily requests to a handful, suggesting real bot suppression.
  • Technical criticism: SHA-256 is ASIC-friendly; JS implementation is inefficient; difficulty calibration is poor; no clear time estimate for users. People quickly wrote native/GPU/OpenCL solvers that bypass the intended cost.

Bot Landscape and Residential Proxies

  • Multiple reports of massive, distributed scraping from residential/mobile IPs, often in Asia/Indonesia, with realistic user agents and low per-IP volume.
  • This traffic harms performance, inflates costs, and threatens businesses that license data and rely on ads/subscriptions.
  • Some blame AI-training scrapers and data brokers, but commenters note attribution is murky and many assumptions are hand-wavy.

Big-Company Crawlers Behaving Badly

  • Complaints about “official” bots (e.g., major clouds and social platforms) ignoring robots.txt, mishandling rate limits (429s), and using deceptive user agents or click IDs to look like humans.
  • Their behavior can resemble a DoS, and explanations from vendors are often vague or withheld as “competitive.”

Mitigation Techniques Beyond Anubis

  • Common tactics: Cloudflare and other CDNs, ASN and subnet blocking, pattern-based blocking via logs, IP reputation/risk databases, JA4/TLS fingerprinting, and browser-fingerprinting tools.
  • Limitations noted: residential proxies can mimic real browsers; sophisticated headless browsers evade many checks; fail2ban and similar tools don’t scale to low-rate, high-IP-count attacks.

Broader Concerns: Anonymity, IDs, and “Proof of Human”

  • Some advocate government digital IDs to fight bots; others argue this enables authoritarian tracking and erodes the right to anonymity.
  • Discussion of CAPTCHAs and “proof-of-human” tests: certain visual patterns may still separate humans from frontier models, but accessibility and false positives (e.g., blind users) remain issues.

Bots on HN and the Social Web

  • Speculation about bot-driven posting and voting on HN; some feel a “vibe change” with faster downvotes and possible shilling.
  • Others think HN is still relatively low-bot compared to large social sites, but acknowledge incentives for automated influence and spam.