How big is YouTube?

Original Article ↗ Hacker News Discussion ↗

Researchers have used a clever sampling technique—treating YouTube’s 64‑bit video IDs like a giant address space and probing it at random—to estimate there are roughly 13 billion public videos on the platform, enabling rough inferences about views, languages, and storage needs at exabyte scale. Commenters probe the statistical assumptions behind this method, debate how ID allocation, autocomplete behavior, and “cheats” might bias results, and compare it to mark‑recapture techniques in ecology. The thread also branches into concerns about opaque recommendation systems, tightened API access after privacy scandals, and whether large platforms should be required to publish aggregate statistics for research on misinformation and media influence.

Estimation Method & Statistical Reasoning

  • Discussion centers on a clever sampling technique to estimate total YouTube videos by probing the ID space and measuring hit-rate.
  • Several comments relate it to mark–recapture, unseen-species problems, Monte Carlo, and the German tank problem.
  • Multiple people argue the estimator does not require video IDs to be uniformly distributed; as long as the probes are uniform over the true ID space, the hit probability equals “assigned IDs / total IDs.”
  • Others initially worry about skewed or structured IDs; replies walk through small toy examples showing clustering doesn’t change the hit probability.
  • One commenter computes an implied standard deviation of under 1% of the estimate, arguing the method is statistically quite precise given the enormous effective sample size.

“Cheats”, Independence, and Biases

  • The main “cheat” is using YouTube search case-insensitivity and OR-chains of random IDs so each query covers ~32,000 real IDs.
  • Concerns: this may oversample IDs containing letters vs digits/symbols, and relies on the search engine to return all matches; it also excludes private and, practically, unlisted videos.
  • Some worry these tricks break independence or make the sample non-random; others think the effect is minor and brute-force would be computationally infeasible.
  • There’s speculation on how YouTube could thwart such studies (random redirects, fake results, hiding searches for random hashes), but several note these countermeasures would be expensive and often detectable.

Implications, Access, and Regulation

  • Some argue platforms this large are quasi-utilities and should be legally required to publish aggregate stats (e.g., dislikes, reach) for democratic oversight.
  • Others counter that YouTube has competitors and regulators should focus on privacy, antitrust, and content rules, not forced data disclosure.
  • Several see API lockdowns (post–Cambridge Analytica and more broadly) as part of the “death of the open web,” and praise scraping-based research as a workaround.
  • EU’s Digital Services Act provisions requiring data access for vetted researchers are highlighted as important but potentially biased by “vetting.”

YouTube Scale, Storage, and Impact

  • Using the estimated video count and rough bitrate/length assumptions, commenters derive multi‑exabyte storage needs, with some arguing actual requirements (multiple codecs, resolutions, replication, originals) likely push YouTube toward tens or even ~100 exabytes.
  • There’s side discussion of codecs (H.264, VP9, AV1, HEVC), original uploads, and whether originals are retained.
  • One link gives Google-wide emissions on the order of 10 million tons CO₂/year, which some find striking for a single company.

User Experience and Recommendations

  • Several users report search and recommendations feeling “shrunk” despite YouTube’s vast size: repetitive home feeds, irrelevant “For You” content, and difficulty discovering the long tail.
  • Workarounds include using filters in search and relying on the subscriptions page; some use scripts to avoid the home feed.
  • Others note that from YouTube’s perspective it is cheaper if everyone watches a small subset repeatedly, creating a tension between discovery and monetization.

Miscellaneous

  • The thread notes the accompanying public stats site and a separate “YouTube dislikes” dataset as valuable research resources.
  • A single extremely conspiratorial comment about an “Alliance” wiping out “communist platforms” is widely mocked, including for eccentric semicolon usage, and used to illustrate how fringe beliefs leak into mainstream discussions.