So you want to scrape like the big boys (2021)
Web scraping at scale divides engineers, entrepreneurs, and site owners between those who see public web data as fair game and those who view large-scale harvesting—especially when it bypasses blocks and ToS—as unethical exploitation. Commenters debate ownership of user-generated content, the role of robots.txt and legal rulings, and whether scraping undermines or enables valuable services such as search, fraud detection, and archiving. The thread also highlights an ecosystem of anti-bot defenses, residential proxies, and scraping SaaS, along with concerns that aggressive bot mitigation harms security research and ordinary users’ privacy.
Ethics of Web Scraping
- Strong split between “scraping is usually unethical” vs. “public data is fair game.”
- Critics see most scraping as entitled, hostile to site owners, and often infrastructure-abusive (hammering sites, bypassing protections, commercial exploitation).
- Supporters focus on user autonomy: once bits reach their computer, they feel free to transform, reformat, or archive them, even if that harms a business model.
- Some compare this to adblocking: users reject the idea they “owe” anyone a business model.
Ownership, Rights, and Intellectual Property
- One camp: if you control access to data, you “own” it and can dictate terms of use; bypassing those terms is unethical regardless of legality.
- Opposing view: putting data on a public website destroys practical exclusivity; ideas like “owning” generic data or unpaid user content are questioned.
- Disagreement over whether web publishers can morally claim ownership of user-submitted content and factual data.
- Some argue “intellectual property doesn’t exist” or applies only to a narrow slice of data.
Consent, Robots.txt, and “Good Neighbor” Behavior
- Many treat robots.txt and bot mitigation as a clear “do not enter” signal; ignoring it and evading blocks is called unethical even if legal.
- Others stress that robots.txt has no inherent legal force and is obscure to most users; they see responsibility on site owners to enforce rate limits and access controls technically.
- Debate on whether noble intentions (e.g., accessibility audits, anti-fascist work) justify scraping without consent.
Legitimate and Harmful Use Cases
- “Positive” uses mentioned: archiving local history, de-duplicating rental listings via RSS, fraud and scam detection, monitoring fake business reviews and listings, security research against phishing and brand impersonation.
- Critics respond that good ends don’t create entitlement to others’ resources; they distinguish small-scale, cooperative scraping from industrial systems designed to defeat defenses.
- Tension highlighted around LLM training and bulk data extraction, seen as an acceleration of the problem.
Legal Landscape and Big Tech Hypocrisy
- Lawyer commenter notes: many dominant tech companies built on scraping but now forbid competitors from scraping them.
- Reference to US cases (e.g., scraping public pages found legal) and to robots.txt-based “consensual” scraping by major search engines.
- Several accuse companies of hypocrisy: aggregating public data themselves (search, radio streams, etc.) while denouncing others who scrape them.
Anti-Bot Tech, Cloudflare, and Security/Privacy
- Security professionals complain anti-bot/CDN services (notably Cloudflare) protect threat actors’ phishing sites and make defensive scraping harder.
- Suggestions for better cooperation: “security bot” programs, faster takedowns, or micro-payment models for controlled access.
- Anti-bot measures noted as privacy-invasive (fingerprinting, port scanning, blocking VMs) and as creating collateral damage for legitimate users and researchers.
- Proposed mitigations include CPU puzzles instead of heavy fingerprinting, though trade-offs for low-powered devices and determined scrapers are acknowledged.
Scraping Industry and Infrastructure
- Commenters describe a mature ecosystem: residential proxy networks, rotating IPs, device farms, SaaS scraping APIs, and anti-anti-bot engineering roles.
- Consensus that the real money is in selling tools and infrastructure to scrapers, not scraping itself.
- Residential proxies often come from user-installed agents or paid home devices; concerns about lax due diligence and possible malware-based networks.