Websites are closing their doors to AI agents

Original Article ↗ Hacker News Discussion ↗

Websites are increasingly blocking AI agents and scrapers, as businesses reliant on ads, upsells, or tightly controlled user flows see automated access as a direct threat to their revenue and infrastructure. Commenters debate whether this is short-sighted in a future where AI agents may become the primary way people shop and browse, warning that sites invisible to agents could lose customers altogether. Proposed paths forward include paid or licensed machine-readable interfaces, micropayments (e.g., HTTP 402-style models), and a new wave of “AI search optimization” that markets directly to agents rather than human users.

Economics and Incentives

  • Many see blocking agents as rational: agents skip ads and upsells, directly attacking ad- and funnel-based business models.
  • Others argue sites are miscalculating: if agents become the main way people interact with the web, sites that block them may “disappear” from users’ choices.
  • Companies also fear easier cross‑shopping and price comparison, which erodes “convenience moats” and dark-pattern UX optimizations.

Will Agents Dominate User Interaction?

  • One side predicts most computer interaction will go through agents; businesses will primarily market to agents, not humans.
  • Skeptics say agents will remain niche: most people don’t trust them, can’t afford powerful ones, won’t pay ongoing costs, and don’t have many tasks that benefit.
  • A counterpoint: if agents stay niche, blocking them is unnecessary; if they become central, blocking them is self‑defeating.

E‑Commerce, Amazon, and Competition

  • Agents could level the field by systematically comparing price, shipping speed, and returns across sites, undermining tactics that rely on user laziness or lock‑in.
  • Some see this as a threat to Amazon; others say Amazon’s logistics, inventory systems, and fast returns remain a decisive moat that agents can’t replicate.
  • Concern: agents may be easily fooled by adversarial marketing, fake reviews, and prompt injections, making “better products” hard to recognize algorithmically.

Technical Blocking and Small Site Impact

  • Bot mitigation via Cloudflare, DataDome, WAFs, and custom rules is widespread; some users report false positives that block even human access.
  • Small sites suffer from heavy scraping (huge log volumes, server overload) and often default to commercial protection services.
  • There is discussion of DIY and open‑source WAFs, proof‑of‑work schemes, behavioral gating, and “human‑like” agent behavior to evade detection.

Alternative Models and Web Fragmentation

  • Proposed compromises:
    • Licensed, paid JSON/XML/API feeds for agents; HTML for humans, possibly hostile to bots.
    • Micropayment schemes (402 Payment Required, Lightning/L402, Cloudflare payment models) to compensate sites per agent request.
    • “AI Search Optimization” (ASO) and direct marketing to LLMs.
  • Several commenters see the situation as another “tragedy of the commons”: open access gets abused (ads, scrapers, DDoS‑like traffic), prompting lock‑in, walled gardens, and increasingly adversarial human–web interactions.