The bot situation on the internet is worse than you could imagine

Original Article ↗ Hacker News Discussion ↗

Anubis Proof-of-Work and User Experience

  • Many commenters couldn’t access the article due to Anubis PoW set at high difficulty: multi-minute or hour-long waits, phones/laptops running hot, high CPU and battery drain.
  • Some see it as effectively blocking the site rather than protecting it; suggestions that at this point you might as well take the site down.
  • Several suspect or joke that it looks like cryptomining or a “honey pot.”
  • Others point out experimental data: at low difficulty in a tar-pit, Anubis reduced hundreds of thousands of daily requests to a handful, suggesting real bot suppression.
  • Technical criticism: SHA-256 is ASIC-friendly; JS implementation is inefficient; difficulty calibration is poor; no clear time estimate for users. People quickly wrote native/GPU/OpenCL solvers that bypass the intended cost.

Bot Landscape and Residential Proxies

  • Multiple reports of massive, distributed scraping from residential/mobile IPs, often in Asia/Indonesia, with realistic user agents and low per-IP volume.
  • This traffic harms performance, inflates costs, and threatens businesses that license data and rely on ads/subscriptions.
  • Some blame AI-training scrapers and data brokers, but commenters note attribution is murky and many assumptions are hand-wavy.

Big-Company Crawlers Behaving Badly

  • Complaints about “official” bots (e.g., major clouds and social platforms) ignoring robots.txt, mishandling rate limits (429s), and using deceptive user agents or click IDs to look like humans.
  • Their behavior can resemble a DoS, and explanations from vendors are often vague or withheld as “competitive.”

Mitigation Techniques Beyond Anubis

  • Common tactics: Cloudflare and other CDNs, ASN and subnet blocking, pattern-based blocking via logs, IP reputation/risk databases, JA4/TLS fingerprinting, and browser-fingerprinting tools.
  • Limitations noted: residential proxies can mimic real browsers; sophisticated headless browsers evade many checks; fail2ban and similar tools don’t scale to low-rate, high-IP-count attacks.

Broader Concerns: Anonymity, IDs, and “Proof of Human”

  • Some advocate government digital IDs to fight bots; others argue this enables authoritarian tracking and erodes the right to anonymity.
  • Discussion of CAPTCHAs and “proof-of-human” tests: certain visual patterns may still separate humans from frontier models, but accessibility and false positives (e.g., blind users) remain issues.

Bots on HN and the Social Web

  • Speculation about bot-driven posting and voting on HN; some feel a “vibe change” with faster downvotes and possible shilling.
  • Others think HN is still relatively low-bot compared to large social sites, but acknowledge incentives for automated influence and spam.