13ft – A site similar to 12ft.io but self-hosted

Original Article ↗ Hacker News Discussion ↗

A self-hosted tool called 13ft, which impersonates Googlebot to strip ads and bypass paywalls on news sites, is drawing interest as 12ft.io and similar services become unreliable or face DMCA pressure. Commenters weigh its technical limits and alternatives (browser extensions, archive sites, library access) against ethical concerns over effectively “stealing” paid journalism, arguing over whether bypassing paywalls is justified by poor user experience, high subscription costs, and Google’s role in enabling cloaked content for its crawler. Broader proposals range from micropayments and cross-publication bundles to treating news access as a utility funded via taxes or ISP fees.

What 13ft Is and How It Works

  • Self‑hosted clone of 12ft.io that fetches pages with a Googlebot‑like user agent and strips ads/popups/paywalls.
  • Some like using it as a shareable “clean” article relay for friends or across devices.
  • Author states it was a quick proof‑of‑concept, not meant to be perfect.

Effectiveness and Technical Limitations

  • Mixed reports: works for some sites (including NYT for some users), but fails or only works once for others.
  • Many note big publishers likely validate Googlebot via IP / DNS or Cloudflare/WAF rules, making simple UA spoofing unreliable.
  • Some suspect sites asynchronously blacklist IPs that fake Googlebot.
  • Others are surprised any site still trusts the UA string alone.

Alternatives and Browser‑Side Approaches

  • Many use archive.org / archive.today / archive.is / archive.ph to bypass paywalls and preserve content.
  • Popular tools: Bypass Paywalls Clean (now DMCA‑targeted and mirrored in Russia), user‑agent switcher extensions, Requestly‑style header rewriting, Firefox about:config overrides.
  • uBlock Origin, NoScript, Pi‑hole, and similar are recommended for ads/popups and tracking, but they don’t usually defeat hard paywalls.
  • iOS limitations (WebKit requirement, weaker extension model) push some toward server‑side or shortcut‑based workarounds.

Ethical and Legal Debate

  • Strong split between those who see paywall bypassing as close to theft and those who see it as fair use given:
    • Poor UX (ads, trackers, popups, hard‑to‑cancel subs).
    • Desire to read only a handful of articles across many outlets.
    • Publishers whitelisting search bots while blocking regular users.
  • Some liken it to piracy; others argue no “taking” occurs, only copying, and point to publishers’ own cloaking as a “bait and switch.”

Search Engines, Cloaking, and Google’s Role

  • Several argue that showing free content to Googlebot but paywalled content to users should violate search policies (cloaking) and that paywalled content shouldn’t be indexed.
  • Others defend current practice as necessary to fund journalism while keeping articles discoverable.
  • Discussion extends into Googlebot IP verification, Cloudflare behavior, and antitrust concerns around exclusive data access (e.g., Reddit deals).

Business Models, Pricing, and Access

  • Many say they pay for a few core publications but cannot afford or justify dozens more just for occasional articles.
  • Strong interest in:
    • Micropayments / pay‑per‑article.
    • Netflix/Spotify‑style news bundles.
    • Library‑mediated access (PressReader, NYT day‑passes), which some report as working well in various countries.
  • Some propose universal syndication funded via taxes or ISP‑level fees; others worry about giving ISPs more power.