Why wordfreq will not be updated
An open-source project that measured how often words appear in real-world text is being frozen because its creator no longer believes it’s possible to get a clean, human‑only sample of language from the modern web. Commenters explore how large language models and SEO spam have “polluted” online text, drawing analogies to post‑nuclear “low‑background steel,” and note that major platforms are also locking down their data to prevent unrestricted AI scraping. The thread broadens into concerns about feedback loops where AI‑generated content trains future AIs and shapes human writing, the decline of traditional NLP research, and emerging efforts to curate pre‑AI or explicitly human corpora.
Impact of generative AI on language data
- Many see the open web as increasingly “polluted” by LLM‑generated text, making it impossible to build corpora that represent human language use post‑2021.
- Feedback loops worry people: models overuse certain words (“delve”, “seamless”, etc.), that usage spreads to humans, then back into model training, amplifying quirks and flattening style.
- Some argue this is just another form of language evolution; others fear “model collapse” where both AI and human language converge into generic, low‑information prose.
Data access and enclosure
- Past key text sources (Twitter/X, Reddit) have locked down or monetized APIs, often explicitly to sell data for AI training.
- This dual pressure—AI slop on the open web plus paywalled “clean” data—makes reproducing projects like wordfreq in the old way infeasible.
- Several commenters liken this to low‑background steel or pre‑atomic datasets: pre‑AI text becomes a scarce, valuable resource.
Possible responses and alternatives
- Proposals:
- Maintain curated whitelists of human‑only sources (not publicly listed).
- “Vintage” or “handmade” data services; scanning old books, microfiche, pre‑LLM archives.
- Badges or metadata marking non‑AI content (with skepticism about enforceability).
- Fork wordfreq anyway to study AI’s impact on language rather than avoid it.
- Others retreat to smaller or older media: RSS, self‑hosted blogs, niche forums, books printed before ~2020, offline apps and “private internets”.
Detecting AI vs human text
- Ideas include using word‑frequency fingerprints, perplexity, or GAN‑style discriminator models to spot LLM output.
- Counter‑arguments: detection is a moving target; models can be tuned to evade tests; statistical tests are noisy and confounded by genre, topic, and natural language change.
Broader concerns and attitudes
- Strong pessimism about “enshittification” of the web, search, commerce, and social feeds, with AI seen as an accelerant atop long‑standing SEO/content‑farm problems.
- Others report encountering less obvious AI slop and emphasize LLMs as practical tools that enable tasks (e.g., obscure software configuration) that would otherwise be prohibitively time‑consuming.
- Debate is polarized: some view anti‑AI sentiment as Luddite or ego‑driven; others see AI as inherently political, reshaping jobs, truth, and control over culture and data.