The Pile is a 825 GiB diverse, open-source language modelling data set (2020)

Dataset status & related corpora

• The Pile is described as “old” and apparently no longer directly downloadable from the official link; people point to torrents/magnet links and hard‑drive swaps as the main distribution.
• Several commenters highlight newer or alternative datasets:
  • The Stack v2 (code-focused) with 67 TB raw, cleaned down to ~3 TB (900B tokens).
  • Dolma and other mixtures of curated sources + filtered Common Crawl.
• There is mention of an in‑progress “Pile v2” using only permissively licensed data, seen by some as a downgrade in capability but a legal necessity.

Content composition & scale

• The Pile mixes many curated sources; key controversial components include:
  • “Books3”: a dump of a private ebook tracker with large amounts of copyrighted books.
  • “OpenWebText2”: web pages linked from highly upvoted Reddit posts.
  • Subtitles from “opensubtitles,” which likely include full copyrighted scripts.
• Some see 825 GB as surprisingly small; others note it feels big once you try to download or host it.

Copyright, fair use & legality

• Strong disagreement over legality:
  • One side argues training on copyrighted data is or should be fair use, and that models are transformative, small compared to training data, and rarely regurgitate long passages.
  • The other side argues:
    • Assembling and redistributing raw corpora of pirated books, subtitles, etc. is straightforward copyright infringement.
    • Fair use can’t be self‑declared; courts apply a four‑factor test (purpose, nature, amount, market harm).
    • These datasets likely fail on “amount” (entire works) and “market effect” (models competing with authors and artists).
• Distinction is made between:
  • Using works privately to train a model vs.
  • Publicly distributing those works in bulk datasets.
• Some see this as analogous to torrent sites; others compare to Google Books and previous fair‑use wins.

Ethics, labor, and impact on creators

• Many posts frame dataset creators and AI companies as “laundering” massive copyright violations and exploiting unpaid creative labor.
• Others argue information should be shared broadly, intellectual property is overextended, and society should embrace the benefits even if traditional business models suffer.
• Concerns include economic displacement of writers/artists, style imitation, and potential misuse of training data (e.g., CSAM in large image datasets).