Claim: Private GitHub repos included in AI dataset

Original Article ↗ Hacker News Discussion ↗

Scope of the Issue (What The Stack Is Using)

  • The Stack v2 is built from the Software Heritage archive, which mirrors “publicly available” source code from GitHub and many other forges.
  • Software Heritage’s own policy (as quoted) is to archive everything public, without checking licenses, leaving license compliance to downstream users.
  • The Stack reportedly excludes some GPL/AGPL-licensed repos but includes many unlicensed ones and some with non‑permissive terms.

Private vs Public Repos: Conflicting Claims

  • Several commenters say they found old, now‑deleted or private repos in The Stack and strongly believe they were always private.
  • Many more report that only repos that were ever public appear; repos that have “always been private” do not.
  • Some cases where people thought a repo was always private were shown via Software Heritage or GitHub Archive data to have been public at some point.
  • A GitHub employee (posting informally) suggests GitHub’s role is limited: Software Heritage archives public repos that may remain even after deletion on GitHub.
  • Others note that private repos used to require paid plans and that brief accidental public exposure is common, which may explain memories conflicting with archival evidence.
  • Several commenters explicitly call the “GitHub leaked private repos” claim unproven and likely misremembered; others insist they and colleagues see genuinely private repos but provide no verifiable examples. Status: unclear.

Verification Tools and Forensics

  • Suggested checks:
    • Software Heritage search to see if a repo was ever archived.
    • GitHub Archive / ClickHouse queries for PublicEvent to detect when a private repo was made public.
    • Wayback Machine snapshots, where available.
  • GitHub’s own audit log only goes back ~6 months, so it cannot resolve older cases.

Licensing and Legality Debates

  • Strong concern that Software Heritage and The Stack ingest code regardless of license, including proprietary and unlicensed code that is still fully copyrighted.
  • Debate over whether making a repo public creates an “implied license” for crawling, archiving, and search; some cite US case law and fair use, others argue this would not hold, especially under French law (for Software Heritage).
  • Questions raised about whether redistributing a training dataset without preserving license notices violates copyleft or attribution requirements.
  • Some argue any human‑permissible “reading and learning” should also be permissible for AI; others respond that this sidesteps license terms that explicitly restrict uses such as training, military use, etc.

Consent, Ethics, and Opt‑Out Model

  • Heavy criticism of opt‑out consent: “consent is not opt‑out,” especially for commercial AI.
  • The opt‑out interface is described as patronizing and slow; at least one early opt‑out issue reportedly remained unresolved.
  • Several people say that making a repo non‑public or deleting it should be treated as a strong “do not use” signal for future dataset versions, even if technically public in the past.

Software Heritage Practices and Resistance

  • One person shares having received an email that their self‑hosted forge would be periodically archived; they objected on copyright grounds and got the request canceled, but were told others could still “Save code now.”
  • Others report successfully demanding takedowns of hundreds of repos and/or blocking Software Heritage IPs or user agents.
  • Some call for forges or hosts to block Software Heritage crawlers by default.

Attitudes Toward GitHub, AI, and Risk

  • Many see this as part of a broader pattern: public hosting platforms and AI projects extracting value from user code without meaningful consent or compensation.
  • Others call the whole thing a “nothingburger,” arguing that only ever‑public code is included and that there’s no credible evidence of private repo leaks—while acknowledging the optics and licensing issues remain serious.
  • Suggestions range from encrypting private repos, moving off GitHub, or building end‑to‑end encrypted git services, to “poisoning” public repos as a protest (not elaborated technically).