Many AI safety orgs have tried to criminalize currently-existing open-source AI
Efforts by some AI safety organizations to restrict or even criminalize open‑source AI models are being widely interpreted as a form of regulatory capture that would entrench the power of large tech firms. Commenters debate whether licensing of training data and compute limits will effectively end community‑driven AI development in the West, pushing innovation into less regulated jurisdictions or underground. Alongside this, there is a sharp split between those focused on speculative existential risks from future AGI and those more concerned with current harms like copyright violations, misinformation, and job losses—and whether broad bans on open models are either practical or desirable.
Regulatory capture and motives of “AI safety” orgs
- Many see AI safety groups as de‑facto lobbyists for large incumbents (OpenAI, Microsoft, Google, etc.), using “safety” to justify rules that entrench corporate moats.
- Critics argue these orgs push for bans or criminalization of open models while being quieter about equivalent or greater risks from closed, corporate systems.
- Some commenters suspect coordination with governments and intelligence/military interests; others think most rank‑and‑file safety researchers are sincere but their funders are not.
Data, copyright, and the future of open‑source AI
- A major concern: if training data must always be licensed, open models could become economically impossible, while big tech can afford deals.
- Others counter that:
- FOSS communities will keep training on unlicensed data “in the shadows” (torrent/IPFS/foreign jurisdictions).
- Synthetic or crowd‑sourced “FOSS data” and public‑domain material can partially substitute, though likely with degraded capabilities.
- There’s broad disagreement on whether training on copyrighted content is theft, fair use, or something in between.
Existential risk vs present‑day harms
- One camp focuses on x‑risk: uncontrolled AGI potentially destroying or permanently disempowering humanity; they see openness as making capabilities irreversible.
- A larger portion of the thread is skeptical: sees current models as far from AGI and x‑risk arguments as speculative, assumption‑stacked, or sci‑fi.
- Many argue near‑term harms are more concrete: labor displacement, devaluation of creative work, misinformation, spam, plagiarism‑laundering, biased decision systems, and power concentration.
Bans, thresholds, and enforceability
- Proposals discussed include FLOP‑based thresholds, bans on certain models, or even extreme ideas like globally banning integrated circuits.
- Pushback is strong:
- Enforcement is seen as unrealistic (like drug prohibition or “banning malicious software”).
- Nation‑state competition and cheap compute make unilateral bans unstable; rogue actors or other countries would continue regardless.
- Several note that vague “AI safety” criteria can easily be stretched to censor disfavored speech or protect incumbents.
Value and risks of open‑source models
- Supporters see open models as:
- Essential for democratizing AI, resisting corporate control, and enabling independent research on failure modes.
- Analogous to open cryptography; banning them would harm the West’s competitiveness and simply shift development elsewhere.
- Critics worry open models make uncensored misuse (weapons advice, disinformation, scams) easier and harder to “take back” once released.
Conceptual confusion around “AI safety”
- Commenters note “AI safety” often conflates:
- Content moderation / “safe outputs”.
- Societal harms (jobs, propaganda, copyright).
- Long‑term AGI x‑risk.
- This overloading fuels talking‑past‑each‑other and allows actors to hide profit‑protection behind more legitimate safety concerns.