Classifying all of the pdfs on the internet

Original Article ↗ Hacker News Discussion ↗

An experiment to categorize hundreds of thousands of PDFs from a Common Crawl-derived 8TB corpus has prompted debate over how representative that dataset is of the internet’s actual PDF volume, with estimates ranging from tens to hundreds of terabytes once academic, corporate, and private documents are considered. Commenters highlight both the technical side—using embeddings and LLMs to classify documents via metadata or URLs, and the challenges of reliably extracting structured data like tables—as well as broader concerns about web crawling, data permanence, and the limits of the “right to be forgotten.” Many see the work as a compelling proof of concept, but argue that its title overstates how much of the internet’s PDF content it really covers.

Scale of the PDF Corpus

  • Many argue 8 TB is small relative to “all PDFs on the internet.”
  • Comparisons: Libgen and Anna’s Archive are reported in the tens to hundreds of TB; one estimate for Google Scholar–indexed PDFs alone implies >50 TB.
  • Several commenters with private collections report 10–40 TB+ of PDFs (scientific, manuals, magazines), suggesting total global volume is likely far larger, possibly petabytes when including private documents (invoices, contracts, scans).

Common Crawl and Dataset Limitations

  • Common Crawl typically truncates PDFs at ~1 MiB; the SafeDocs dataset refetched untruncated versions for one snapshot.
  • Some note that this still only covers the “open web,” excludes paywalled and private corpora, and likely misses many large image-heavy PDFs.
  • One commenter points out that the article effectively works on 500k PDFs, not the full corpus, and likely on metadata/URLs rather than full content.

Right to Be Forgotten (RTBF) Debate

  • RTBF is discussed tangentially: some see it as futile once data is online; others clarify that laws target specific service providers and search engines, not “the entire internet.”
  • There is disagreement over whether RTBF aims only to prevent use/storage, or also to remove public searchability of past information.
  • Concerns are raised about RTBF being used by scammers or convicted individuals to bury past misconduct.

Personal Archives and Copyright

  • Multiple users describe large private PDF archives (scientific papers, manuals, historical magazines).
  • Efforts to build public magazine repositories face copyright and DMCA risks; many historical issues are in legal limbo with unclear rights holders.

PDF Extraction, Partitioning, and Embeddings

  • Several are more interested in techniques for robust PDF parsing and data extraction (especially tables) than in size debates.
  • Tools like Aryn and others are mentioned for partitioning PDFs and converting tables into structured data.
  • Commenters highlight that embeddings enable using standard statistical and ML techniques without complex NLP preprocessing.

Critiques of Article Framing

  • Some see the title (“all PDFs on the internet”) as marketing overreach, given the limited corpus and reliance on URLs.
  • Others still find the approach and visualizations valuable as a demonstration of embeddings-based classification at scale.