So you want to scrape like the big boys (2021)

Ethics of Web Scraping

• Strong split between “scraping is usually unethical” vs. “public data is fair game.”
• Critics see most scraping as entitled, hostile to site owners, and often infrastructure-abusive (hammering sites, bypassing protections, commercial exploitation).
• Supporters focus on user autonomy: once bits reach their computer, they feel free to transform, reformat, or archive them, even if that harms a business model.
• Some compare this to adblocking: users reject the idea they “owe” anyone a business model.

Ownership, Rights, and Intellectual Property

• One camp: if you control access to data, you “own” it and can dictate terms of use; bypassing those terms is unethical regardless of legality.
• Opposing view: putting data on a public website destroys practical exclusivity; ideas like “owning” generic data or unpaid user content are questioned.
• Disagreement over whether web publishers can morally claim ownership of user-submitted content and factual data.
• Some argue “intellectual property doesn’t exist” or applies only to a narrow slice of data.

Consent, Robots.txt, and “Good Neighbor” Behavior

• Many treat robots.txt and bot mitigation as a clear “do not enter” signal; ignoring it and evading blocks is called unethical even if legal.
• Others stress that robots.txt has no inherent legal force and is obscure to most users; they see responsibility on site owners to enforce rate limits and access controls technically.
• Debate on whether noble intentions (e.g., accessibility audits, anti-fascist work) justify scraping without consent.

Legitimate and Harmful Use Cases

• “Positive” uses mentioned: archiving local history, de-duplicating rental listings via RSS, fraud and scam detection, monitoring fake business reviews and listings, security research against phishing and brand impersonation.
• Critics respond that good ends don’t create entitlement to others’ resources; they distinguish small-scale, cooperative scraping from industrial systems designed to defeat defenses.
• Tension highlighted around LLM training and bulk data extraction, seen as an acceleration of the problem.

Legal Landscape and Big Tech Hypocrisy

• Lawyer commenter notes: many dominant tech companies built on scraping but now forbid competitors from scraping them.
• Reference to US cases (e.g., scraping public pages found legal) and to robots.txt-based “consensual” scraping by major search engines.
• Several accuse companies of hypocrisy: aggregating public data themselves (search, radio streams, etc.) while denouncing others who scrape them.

Anti-Bot Tech, Cloudflare, and Security/Privacy

• Security professionals complain anti-bot/CDN services (notably Cloudflare) protect threat actors’ phishing sites and make defensive scraping harder.
• Suggestions for better cooperation: “security bot” programs, faster takedowns, or micro-payment models for controlled access.
• Anti-bot measures noted as privacy-invasive (fingerprinting, port scanning, blocking VMs) and as creating collateral damage for legitimate users and researchers.
• Proposed mitigations include CPU puzzles instead of heavy fingerprinting, though trade-offs for low-powered devices and determined scrapers are acknowledged.

Scraping Industry and Infrastructure

• Commenters describe a mature ecosystem: residential proxy networks, rotating IPs, device farms, SaaS scraping APIs, and anti-anti-bot engineering roles.
• Consensus that the real money is in selling tools and infrastructure to scrapers, not scraping itself.
• Residential proxies often come from user-installed agents or paid home devices; concerns about lax due diligence and possible malware-based networks.