Curl-Impersonate

Original Article ↗ Hacker News Discussion ↗

What curl-impersonate actually does

  • Modifies curl’s TLS/HTTP behavior so that, at the HTTP/SSL layer, it matches real browsers’ fingerprints (e.g., Chrome via BoringSSL).
  • “Identical” does not mean byte-for-byte or packet-for-packet (TLS is randomized), but that the observable fingerprint (ClientHello, ciphers, extensions, ALPN, etc.) matches so defenses can’t reliably distinguish it.
  • This is reportedly sufficient to bypass some Cloudflare/WAF bot checks that rely heavily on TLS fingerprints.

Why normal clients look different

  • Command-line tools and basic HTTP libraries typically:
    • Use different TLS stacks (OpenSSL, etc. vs BoringSSL/NSS).
    • Offer fewer cipher suites, extensions, and ALPN options.
    • Omit GREASE and other randomness used by browsers.
  • Result: very different ClientHello fingerprints from mainstream browsers.

Use cases and limits

  • Main use: scraping or API access where curl/python-requests are blocked or heavily challenged, often via WAF presets that treat non-browser fingerprints as bots.
  • Considered a lighter alternative to full headless browsers, which are resource-heavy and fetch unnecessary assets.
  • Some argue serious anti-bot setups also rely on JavaScript checks, behavioral signals, and captchas, so TLS impersonation alone often isn’t enough, but can help for specific API endpoints or token acquisition flows.

Anti-bot arms race

  • Defenders report large-scale abuse: API abuse, scalpers hammering stock-check endpoints, residential proxy botnets, and traffic that looks syntactically “clean.”
  • Simple IP bans or fail2ban rules are described as effective against low-effort worms but inadequate against sophisticated, distributed bots.
  • Techniques mentioned: TLS fingerprinting, header correlation, JS checks, proof-of-work, rate limiting, long-term reputation, forcing auth/verified accounts.
  • Others counter that broad tracking and heavy-handed controls are often unnecessary and privacy-invasive for many threat models.

Related tools and ecosystem

  • Similar ideas exist in Go, Python (using Chromium’s network stack), C#, Rust, and via proxies that rewrap TLS (e.g., JA3/utls-based).
  • Python bindings (curl_cffi) expose a requests-like API backed by curl-impersonate.

Build and maintenance issues

  • Multiple reports that the build system is fragile: autotools + BoringSSL patches, -Werror failures, missing dependencies, and slow builds.
  • Prebuilt binaries and bindings are suggested as the practical way to consume it; the codebase is described as a deliberate “hack” to keep pace with changing browser fingerprints.

Broader concerns about the web

  • Some see tools like this as a symptom of a closing web: increasing reliance on approved clients, device identity, WAFs, and regulatory-driven gating.
  • Others emphasize that escalating controls are also responses to real, large-scale abuse and regulatory pressure, not just corporate tracking motives.