What happened to TheNumbers.com

Original Article ↗ Hacker News Discussion ↗

AI-driven web scraping and prediction-market arbitrage are overwhelming small, data-rich sites like box office database TheNumbers.com, pushing bandwidth and infrastructure costs beyond what hobbyist or lightly funded operators can sustain. Commenters debate whether better caching, CDNs, and static-site rewrites can solve the problem, or whether the core issue is that AI companies and betting platforms externalize costs while monetizing scraped data. Many worry this dynamic will push more valuable information behind paywalls or offline entirely, undermining the open web and volunteer-driven resources.

Multiple overlapping problems

  • Commenters note the article mixes issues:
    – Excessive bot traffic and bandwidth cost
    – Scraping that bypasses paid/licensed access
    – Security risk from attackers probing for early/privileged data
  • Some think this is a niche case (old codebase, niche data, prediction-market angle) rather than a general “AI killed the web” story; others see it as emblematic.

Technical mitigation ideas and limitations

  • Many suggest static-site rebuilds, heavy caching (Varnish/CDNs), or Cloudflare-style protection; some claim this is almost a “weekend project” with modern tools and LLM help.
  • Pushback:
    – Static/caching doesn’t fix the business model if bots extract all value.
    – Cloud/CDN egress can be ruinous under scraper load; examples include tens of TBs of HTML and $1k+ monthly bills for hobby sites.
    – Residential-proxy and headless-browser bots ignore robots.txt and are hard to block.
    – Cache invalidation and dynamic search endpoints are still fundamentally hard.

Economics, business models, and incentives

  • TheNumbers apparently relies more on private data sales than ads; bots don’t click ads or pay.
  • Several expect more content to go behind paywalls or move to hosted data platforms, or just disappear.
  • Some suggest micro-payment or “bots must pay” infrastructure; others are skeptical this can be made to work socially/technically.

Prediction markets as a driver

  • Many see unregulated prediction markets as the key incentive for abusive scraping and even hacking to front-run data releases.
  • Proposed fixes: lock trading before close, or outright ban such markets; counterpoint: competition and users’ indifference to fairness make self-regulation unlikely.

Ethics of scraping and AI training

  • Strong split:
    – One side: if you publish openly, you shouldn’t control who/what consumes it; AI training is just another use.
    – Other side: having free work hoovered into proprietary models feels exploitative and discourages future open contributions, especially when AI returns almost no traffic and imposes real costs.
  • Some discuss new licenses (e.g., GPL/AGPL variants) to restrict training, with skepticism that they’d be enforceable if training is deemed fair use.

Future of the open web

  • Widespread worry that AI scrapers “socialize costs, privatize profits” and accelerate the decline of open, hobbyist resources.
  • Others see AI as a powerful tool that can also help secure and modernize such sites, if maintainers choose to use it.