Claim: Private GitHub repos included in AI dataset

Original Article ↗ Hacker News Discussion ↗

Claims that private or deleted GitHub repositories have been included in “The Stack,” an AI training dataset built from the Software Heritage code archive, are prompting scrutiny of how the data was collected. Many developers who checked the dataset report seeing only repositories that were public at some point, suggesting misremembered privacy settings rather than a confirmed GitHub breach, but the episode highlights broader worries about opt‑out consent, copyright compliance, and the legality of using unlicensed or copyleft code for AI training. Several commenters also criticize Software Heritage and AI projects for archiving and reusing publicly reachable code without clear, explicit permission or easy ways to exclude it.

Scope of the Issue (What The Stack Is Using)

  • The Stack v2 is built from the Software Heritage archive, which mirrors “publicly available” source code from GitHub and many other forges.
  • Software Heritage’s own policy (as quoted) is to archive everything public, without checking licenses, leaving license compliance to downstream users.
  • The Stack reportedly excludes some GPL/AGPL-licensed repos but includes many unlicensed ones and some with non‑permissive terms.

Private vs Public Repos: Conflicting Claims

  • Several commenters say they found old, now‑deleted or private repos in The Stack and strongly believe they were always private.
  • Many more report that only repos that were ever public appear; repos that have “always been private” do not.
  • Some cases where people thought a repo was always private were shown via Software Heritage or GitHub Archive data to have been public at some point.
  • A GitHub employee (posting informally) suggests GitHub’s role is limited: Software Heritage archives public repos that may remain even after deletion on GitHub.
  • Others note that private repos used to require paid plans and that brief accidental public exposure is common, which may explain memories conflicting with archival evidence.
  • Several commenters explicitly call the “GitHub leaked private repos” claim unproven and likely misremembered; others insist they and colleagues see genuinely private repos but provide no verifiable examples. Status: unclear.

Verification Tools and Forensics

  • Suggested checks:
    • Software Heritage search to see if a repo was ever archived.
    • GitHub Archive / ClickHouse queries for PublicEvent to detect when a private repo was made public.
    • Wayback Machine snapshots, where available.
  • GitHub’s own audit log only goes back ~6 months, so it cannot resolve older cases.

Licensing and Legality Debates

  • Strong concern that Software Heritage and The Stack ingest code regardless of license, including proprietary and unlicensed code that is still fully copyrighted.
  • Debate over whether making a repo public creates an “implied license” for crawling, archiving, and search; some cite US case law and fair use, others argue this would not hold, especially under French law (for Software Heritage).
  • Questions raised about whether redistributing a training dataset without preserving license notices violates copyleft or attribution requirements.
  • Some argue any human‑permissible “reading and learning” should also be permissible for AI; others respond that this sidesteps license terms that explicitly restrict uses such as training, military use, etc.

Consent, Ethics, and Opt‑Out Model

  • Heavy criticism of opt‑out consent: “consent is not opt‑out,” especially for commercial AI.
  • The opt‑out interface is described as patronizing and slow; at least one early opt‑out issue reportedly remained unresolved.
  • Several people say that making a repo non‑public or deleting it should be treated as a strong “do not use” signal for future dataset versions, even if technically public in the past.

Software Heritage Practices and Resistance

  • One person shares having received an email that their self‑hosted forge would be periodically archived; they objected on copyright grounds and got the request canceled, but were told others could still “Save code now.”
  • Others report successfully demanding takedowns of hundreds of repos and/or blocking Software Heritage IPs or user agents.
  • Some call for forges or hosts to block Software Heritage crawlers by default.

Attitudes Toward GitHub, AI, and Risk

  • Many see this as part of a broader pattern: public hosting platforms and AI projects extracting value from user code without meaningful consent or compensation.
  • Others call the whole thing a “nothingburger,” arguing that only ever‑public code is included and that there’s no credible evidence of private repo leaks—while acknowledging the optics and licensing issues remain serious.
  • Suggestions range from encrypting private repos, moving off GitHub, or building end‑to‑end encrypted git services, to “poisoning” public repos as a protest (not elaborated technically).