It took a year to ship WebAssembly in Anubis

Original Article ↗ Hacker News Discussion ↗

Proof-of-work challenges like Anubis are emerging as an alternative to CAPTCHAs for throttling abusive web scraping and bot traffic, now using WebAssembly and memory‑hard functions (Argon2) to better align with modern browsers and hardware. Commenters debate whether this approach meaningfully deters large, well-funded scrapers or mainly penalizes ordinary users—especially those on older devices, with JavaScript/WASM disabled, or strong privacy setups. The exchange also highlights broader concerns: ecological and economic costs of “wasted” compute, compatibility trade-offs across browsers and devices, and the limits of traditional tools like IP blocking and robots.txt in an era of residential proxy networks and aggressive AI-driven scraping.

Role and economics of proof‑of‑work

  • PoW is framed as an economic throttle: not to eliminate scraping, but to make abusive traffic too expensive.
  • Some see cycles as wasteful, especially given climate concerns; others say the cost is justified if it curbs denial‑of‑wallet–style scraping.
  • Debate over whether memory‑hard functions (argon2id) meaningfully constrain well‑funded AI scrapers, or mainly hurt small “skiddy” bots and ordinary users.

Impact on users, devices, and privacy

  • Users report mobile devices getting hot or taking noticeable time to solve challenges; desktops and VPSes handle them quickly.
  • Configuration matters: low difficulty and cookie reuse can make PoW nearly invisible for normal browsing, but harsh settings can break old or low‑end devices.
  • People who disable JS, WASM, or cookies for privacy often find Anubis unusable; others argue that breaking “normal” web assumptions leads to such friction.
  • Concerns raised that service‑worker precomputation and IP‑binding can conflict with privacy tools, dynamic IPs, and network changes.

Effectiveness against bots and scrapers

  • Supporters say real deployments show large reductions in abusive bot traffic, especially from rotating‑IP scrapers hammering search or git web UIs.
  • Skeptics argue that the computational cost per challenge is trivial for serious scrapers and that they can adapt (reuse sessions, change strategies).
  • PoW is presented as one layer in defense‑in‑depth, complementing traditional techniques when IP blocking fails due to vast proxy networks.

Robots.txt, “good bots,” and static content

  • Some ask why not rely on robots.txt; others reply that only “good” bots obey it, so restrictive policies give little protection.
  • Static‑looking pages can still incur real CPU and bandwidth costs, and many deployments cannot use external CDNs by policy.
  • There is tension between wanting to block “dishonest” bots and not unduly harming legitimate archiving or research crawlers.

Implementation details: WASM, JS, and Rust tooling

  • Anubis uses highly optimized WASM plus a wasm‑to‑JS fallback for environments without WASM, with substantial effort spent on backward compatibility (old Chrome, Firefox ESR, smart TVs).
  • Extensions already exist that accelerate solving via WASM or WebGPU; argon2 is itself GPU‑accelerable, complicating “GPU‑proof” claims.
  • Rust tooling issues around wasm32-unknown-unknown, MVP vs non‑MVP features, and build-std are discussed; some suggest alternate targets like wasm32v1-none.

Ideas for “useful” or monetized work

  • Multiple commenters suggest replacing pure waste with:
    • Micro‑payments or token systems where sites earn from PoW.
    • Exchanging PoW credits across sites (jokingly “reinventing crypto”).
    • Redirecting PoW into socially useful tasks (e.g., scientific computing, distributed fuzzing), though task granularity makes this hard.