An update on residential proxies and the scraper situation

Original Article ↗ Hacker News Discussion ↗

Residential proxies & legality

  • Many see residential proxies as “botnets in plain sight”: SDKs in free apps, VPNs, and media boxes quietly turn user devices into exit nodes.
  • Dispute over legality: some argue it’s not clearly illegal without explicit bans on such SDKs; others say there’s no informed consent and it should be treated as malware-like.
  • Debate over whether app‑store bans (e.g., on SDKs like NetNut’s) are legitimate security measures or monopoly abuse.

Scraping intensity and impact

  • Core complaint: massive, poorly behaved scraping overwhelms sites, especially dynamic endpoints (search, diffs, paginated lists), effectively acting as DDoS.
  • Residential proxies plus rapid IP rotation make IP‑based blocking ineffective and risk collateral damage to innocent users.
  • Some think this is largely driven by AI training demand; others argue it looks more like generic or even malicious traffic, not rational data collection.

Shared crawls and data access

  • Several propose “better Common Crawl” or per‑site downloadable archives/ZIPs so scrapers don’t need to hammer sites.
  • Counterpoint: projects that already publish full free dumps (Wikipedia, OpenStreetMap, etc.) are still heavily scraped; many bots ignore these channels.
  • Common Crawl contributors note datasets are large but accessible, with indices and recrawl logic; usage is meaningful but not ubiquitous.

Poisoning and data quality

  • A community actively tries to “poison” scraper/LLM training data.
  • Critics say this is mostly symbolic: the web is already noisy, and straightforward filtering and cross‑checking can largely neutralize such efforts.

Mitigations: PoW, captchas, and paywalls

  • Proof‑of‑work systems (e.g., Anubis) are seen by some as the least-bad defense, preferable to captchas and compatible with stateless HTTP.
  • Others report usability problems, privacy concerns, and argue PoW doesn’t scale: attackers can run optimized native code or leverage botnets; browsers are at a disadvantage.
  • Ideas surfaced: HTTP 402 with per‑request payments (possibly via Lightning), karma/identity‑based rate tiers, or site‑wide micropayments. Skepticism centers on friction, economics, and regulation.

Open web tensions

  • Fear that anti‑scraping measures will hurt the open web, block benign crawlers/archivers, and entrench dominant players (Google, Cloudflare).
  • Some praise residential proxies as tools to access data behind corporate walls; others call them an untrustworthy scourge undermining small sites.

Operational tactics & local defenses

  • Operators discuss firewalls, blacklist sharing, fail2ban‑like setups, and traffic monitoring on home routers to spot compromised IoT devices.
  • IP‑range blocking works better in IPv6 by blocking larger prefixes, but widespread CGNAT and proxy churn complicate attribution and fairness.