Web Search API

Original Article ↗ Hacker News Discussion ↗

Cloudflare’s new Web Search API, which resells search from providers like Ceramic, Exa and Linkup for AI agents, is raising questions about whether it adds enough value compared to integrating with those services directly. Supporters point to easier billing, procurement and integration for organizations already standardized on Cloudflare, while critics worry about vendor lock-in, Cloudflare’s growing role as an internet gatekeeper, and potential antitrust or privacy implications given its bot-blocking and intelligence-sharing concerns. The thread also compares pricing and quality with alternatives such as Brave, Kagi, Serper, SearXNG and self-hosted scrapers, highlighting that reliable, affordable web search for LLMs remains a hard and often expensive problem.

Value Proposition of Cloudflare’s Web Search API

  • Many see it as a convenience layer: single integration, billing, limits, and failover across multiple search providers, especially for orgs that already standardize on Cloudflare.
  • Others argue that code agents and direct API use make this abstraction unnecessary, and view it as another attempt to “sit in the middle” and extract rent.
  • For existing Workers/Pages users, the appeal is low-latency, same-ecosystem access to search for agents without adding vendors.

Trust, Centralization, and Power Concerns

  • Some commenters trust Cloudflare as “neutral infrastructure” with strong engineering; others highlight reports of upselling/pressure to upgrade and outages tied to billing/risk systems.
  • There’s extended debate about Cloudflare and other US cloud providers’ exposure to surveillance (FISA/702, NSA).
    • One side: everyone leaks metadata anyway, so Cloudflare is not uniquely bad.
    • Other side: using a TLS-terminating MITM like Cloudflare increases attack and surveillance surface beyond simple hosting.

Bots, Crawlers, and Gatekeeping

  • Concern that Cloudflare is both selling bot protection and monetizing access via “verified bots” and now search APIs.
  • Some fear an anticompetitive pattern: block/hamper generic crawlers, bless/monetize “verified” ones, then charge for access.
  • Others counter that reputable crawlers and “verified bots” are different from abusive scrapers and that operators still control what to block.

Providers, Pricing, and Quality

  • Underlying providers (Ceramic, Exa, Linkup) have very different price points ($0.25–$7 per 1,000 requests), raising questions about quality vs cost.
  • Reports that some “AI-first” search providers return poor or stale results; one user found Ceramic couldn’t even find its own domain or recent Cloudflare docs.
  • Links shared to third-party benchmarks comparing web search APIs; search quality seen as highly use-case-dependent.

Zero Data Retention and Terms of Use

  • Confusion over “Zero Data Retention”: Cloudflare advertises it, but provider docs sometimes say otherwise or charge extra for ZDR; one Exa entry is flagged as “No ZDR”.
  • Commenters note that even with ZDR at the provider, requests still appear in Cloudflare Gateway logs.
  • Several are worried about strict ToS from providers (especially Ceramic) that forbid storing, aggregating, or resyndicating results, limiting agent transcripts and long-lived caches.

Alternatives and DIY Approaches

  • Mentioned alternatives: Brave Search API, Kagi, Serper, SearXNG, self-hosted Firecrawl, Jina, SerpAPI-like services, and Gemini Flash Lite 2.5 search (seen as cost-effective but being constrained/deprecated in places).
  • Some run local or self-hosted gateways (e.g., SearXNG, hister) to avoid vendor lock-in and ToS constraints, accepting more setup in exchange for control.