The Pile is a 825 GiB diverse, open-source language modelling data set (2020)

Original Article ↗ Hacker News Discussion ↗

An influential 825 GiB text corpus for training language models, The Pile, is attracting renewed attention as newer datasets like The Stack v2 emerge and as its original download links disappear, pushing access toward torrents. Commenters highlight that The Pile includes controversial components such as the pirated Books3 and large web scrapes, raising sharp legal and ethical questions around copyright, “fair use,” and whether redistributing such corpora is effectively piracy. The exchange reflects a broader struggle over who should control and profit from the human-created data that powers AI models, and whether future datasets will (or even can) be limited to permissively licensed sources.

Dataset status & related corpora

  • The Pile is described as “old” and apparently no longer directly downloadable from the official link; people point to torrents/magnet links and hard‑drive swaps as the main distribution.
  • Several commenters highlight newer or alternative datasets:
    • The Stack v2 (code-focused) with 67 TB raw, cleaned down to ~3 TB (900B tokens).
    • Dolma and other mixtures of curated sources + filtered Common Crawl.
  • There is mention of an in‑progress “Pile v2” using only permissively licensed data, seen by some as a downgrade in capability but a legal necessity.

Content composition & scale

  • The Pile mixes many curated sources; key controversial components include:
    • “Books3”: a dump of a private ebook tracker with large amounts of copyrighted books.
    • “OpenWebText2”: web pages linked from highly upvoted Reddit posts.
    • Subtitles from “opensubtitles,” which likely include full copyrighted scripts.
  • Some see 825 GB as surprisingly small; others note it feels big once you try to download or host it.

Copyright, fair use & legality

  • Strong disagreement over legality:
    • One side argues training on copyrighted data is or should be fair use, and that models are transformative, small compared to training data, and rarely regurgitate long passages.
    • The other side argues:
      • Assembling and redistributing raw corpora of pirated books, subtitles, etc. is straightforward copyright infringement.
      • Fair use can’t be self‑declared; courts apply a four‑factor test (purpose, nature, amount, market harm).
      • These datasets likely fail on “amount” (entire works) and “market effect” (models competing with authors and artists).
  • Distinction is made between:
    • Using works privately to train a model vs.
    • Publicly distributing those works in bulk datasets.
  • Some see this as analogous to torrent sites; others compare to Google Books and previous fair‑use wins.

Ethics, labor, and impact on creators

  • Many posts frame dataset creators and AI companies as “laundering” massive copyright violations and exploiting unpaid creative labor.
  • Others argue information should be shared broadly, intellectual property is overextended, and society should embrace the benefits even if traditional business models suffer.
  • Concerns include economic displacement of writers/artists, style imitation, and potential misuse of training data (e.g., CSAM in large image datasets).