Web Scraping in Python – The Complete Guide

Original Article ↗ Hacker News Discussion ↗

Web scraping in Python is seen as powerful but increasingly complex, with practitioners leaning on tools like Playwright, Scrapy, requests-cache and third‑party proxy/CAPTCHA services to handle JavaScript-heavy pages, rate limits and bot defenses. Commenters debate Python versus Node.js for scraping, generally favoring Python for its string processing, data tooling and mature frameworks, while noting that browser automation is often overused compared to lighter HTML and API-based approaches. Many emphasize architectural best practices—such as separating crawling from extraction, caching raw pages, and treating scraping as an ETL/ELT pipeline—as well as emerging uses of LLMs to generate or adapt scrapers rather than to parse every page directly.

Browser-based scraping tools

  • Strong enthusiasm for Playwright as a modern, robust browser automation tool (seen as “selenium plus more”), with code generation and device profiles as standout features.
  • Shot-scraper (CLI wrapper around Playwright) praised for convenience and integration with Mozilla Readability, but noted as CPU-heavy and inefficient when invoked thousands of times; for large runs, users suggest writing raw Playwright code.
  • Some confusion over Playwright docs: the intro emphasizes pytest integration, which misleads users who just want the library.

Python vs Node.js and other languages

  • Python’s popularity is attributed to:
    • Long-standing ecosystem (e.g., BeautifulSoup, lxml, Scrapy).
    • Easier string processing and data reshaping vs JS.
    • Simpler synchronous mental model vs JS async.
    • Easy integration with downstream analysis stacks (pandas, DBs).
  • Node/JS is seen as having ergonomic DOM-like APIs but weaker stdlib for text processing.
  • Perl and Ruby are cited as very effective for heavy text-oriented scraping.

Scraping infrastructure & services

  • Multiple proxy + rendering SaaS options mentioned: ScraperAPI, ScrapingBee, Scraping Fish, Apify, Urlbox; users report mixed reliability and cost tradeoffs.
  • Some users factor out proxy/rate-limiting/session logic into standalone proxy-manager services to keep scraper code simple.
  • cloudscraper + proxy lists + threading reported to achieve high request throughput.

Anti-bot defenses, CAPTCHAs, and evasion

  • Scraping is described as far from “dead,” but harder: Cloudflare, Akamai, DataDome, CAPTCHAs, auth walls.
  • Tactics shared:
    • Mobile-phone or residential IPs (including routing via home connection/CGNAT).
    • API reverse-engineering and JSON/LD+JSON/OpenGraph extraction.
    • CAPTCHA-solving services (e.g., 2captcha) or local AI-based solvers.
    • Careful experimentation with rates, headers, and behavior to mimic humans.
  • Some argue bypassing big WAFs (Cloudflare/AWS WAF/Akamai) is nearly impossible; others claim success with impersonation tools and mobile IPs.

Design patterns: crawling, caching, and ETL

  • Strong consensus: separate crawling (HTML acquisition) from scraping (data extraction).
    • Store raw HTML or cached responses (e.g., requests-cache, S3, SQLite) so extractors can be iterated without re-crawling.
  • This is framed as an instance of ETL/ELT best practices: land raw data first, then transform.
  • Simple caching wrappers around HTTP clients are highly recommended even during early experimentation.

LLMs and automation

  • Some are building systems where LLMs generate or adapt scraper code and strategies, rather than using LLMs for every extraction (too slow/expensive).
  • Others experiment with using LLMs on screenshots/DOM, but note context-size and robustness issues; LLM-generated regex and traditional parsing remain attractive.

Ethics, impact, and site perspectives

  • A site operator asks scrapers to use consistent User-Agent strings so traffic can be managed and load-balanced.
  • Another user criticizes anti-scraping solutions like DataDome for degrading normal browsing (especially for lightweight clients) while likely not stopping serious scrapers.

Miscellaneous tips and critiques of the guide

  • Tips: use robots.txt and sitemaps; pandas read_html can simplify table extraction; extruct can pull structured metadata.
  • Some see BeautifulSoup + lxml performance debates as irrelevant because network time dominates.
  • Several commenters call the “complete guide” shallow and largely a vehicle to promote a specific proxy/scraping service.