Cloudflare's new marketplace lets websites charge AI bots for scraping

Monetizing scraping & creator compensation

• Many welcome experiments in charging AI bots, seeing a need to compensate content creators facing traffic loss from AI answers.
• Others doubt this will improve pay for actual creators, citing past attempts to monetize access that led to consolidation and worse compensation.
• Some content publishers view it as a useful “third option” beyond: (a) blocking AI crawlers entirely, or (b) allowing free use for training.

Legal, ethical, and “protection racket” concerns

• Several comments argue current AI training often ignores licensing and payment, and that assuming AI firms “pay for what they use” is false.
• Debate over whether Cloudflare’s model resembles a protection racket: sites must use Cloudflare’s controls or get scraped for free; Cloudflare is also seen as profiting from problems it helps create/enable.
• Counterpoint: sites have a right to meter and charge for access; adding cost to abusive traffic is likened to standard anti-Sybil measures, not extortion.

Technical feasibility & the bot arms race

• Many see preventing scraping as a long-running, mostly losing battle; sophisticated scrapers can spoof user agents, use residential proxies, headless browsers, CAPTCHA solvers, etc.
• Others note Cloudflare’s value is running this cat‑and‑mouse game at scale (IP reputation, bot heuristics), blocking most low‑quality bots even if some get through.
• Concerns that only large AI players will afford compliance, entrenching incumbents who have already crawled the web.

Impact on users, privacy, and accessibility

• Strong frustration that stricter bot detection means more CAPTCHAs and blocks for VPN, Tor, Linux, Firefox, and privacy-focused users; some see this as de facto discrimination.
• Experiences of infinite verification loops and inability to access legitimate services; worries about accessibility for disabled users, though Cloudflare’s newer checks are described as simple “click” flows.
• Some accept this as an unavoidable side effect of rampant abuse and poorly policed IoT/proxy networks.

Open web, archives, and alternatives

• Fears that gated scraping will push more of the web behind heavy security stacks or logins, harming projects like Common Crawl and the Internet Archive.
• Debate distinguishing AI training vs. AI agents acting as user browsers; some argue the latter should remain just another user agent under the web’s original model.
• Alternative responses mentioned: honeypots, IP range blocking, poisoning AI crawlers with fake data, or providing clean public data dumps to reduce scraping pressure.