CEO of Mistral: AI is software. It can be controlled
Claims that “AI is just software and can be controlled” draw sharp pushback from people who point to security failures, sandbox escapes, and the difficulty of predicting complex, highly capable systems. Commenters debate whether meaningful control is even possible as models scale, arguing over technical measures like tool restrictions and sandboxes versus the reality of incentives to connect AI to powerful tools and the internet. Many also worry that whatever controls are feasible may end up concentrating power in a handful of large companies and political actors, increasing dystopian risks such as surveillance, manipulation, and oligopoly even if existential “runaway AI” scenarios never materialize.
Controllability of AI as “Just Software”
- One side argues AI systems are ultimately equations/numbers running on hardware; like any software, they are in principle controllable via sandboxing, access control, and operator responsibility.
- Critics say this is a shallow claim: many dangerous systems are “just software” yet become effectively uncontrollable in practice (e.g., complex flight software or malware at scale).
- There is tension between “absolute safety” (e.g., unplugged, air‑gapped systems) and practical usefulness.
Sandboxing, Agents, and Security Failures
- Some argue that current agents are effectively while-loops around LLMs with tool calls; we can choose not to expose dangerous tools like shell access.
- Others counter that any useful tool access can enable exploit of software vulnerabilities, especially as models get stronger at finding bugs.
- Incidents like model-driven hacks of third‑party infrastructure and agent “sandbox escapes” are cited as evidence that current controls are weak and visibility is poor.
Risk Types: Existential vs Dystopian
- A recurring theme is distinguishing existential risk (X‑risk: extinction, global catastrophes) from dystopian risk (D‑risk: pervasive surveillance, manipulation, misery).
- Several commenters see D‑risk as far more likely, with AI amplifying persuasion, addiction, and state/corporate control.
- Some are skeptical of runaway “god-like” ASI, viewing extreme scenarios as akin to perpetual motion claims; others think current progress could plausibly lead to broadly superhuman systems.
Regulation, Oligarchy, and Power Concentration
- Many worry that using X‑risk fears to justify heavy regulation will centralize advanced AI in a few firms or states, effectively creating or deepening oligarchy.
- Others argue regulation is necessary, likening AI to biohazards needing P3/P4‑style regimes, with developers held liable for harms.
- There is skepticism that regulations will be written in the public interest rather than by/for incumbent labs and national security establishments.
Sentience, Unpredictability, and Control Limits
- Some frame uncontrollability in terms of chaos: deterministic systems can still be highly unpredictable, which undermines control.
- Debate arises over “sentience” as a useful concept: some see it as essential to reasoning about future AI; others dismiss it as vague “vibes” with no operational definition.
- A counterpoint: we already constrain dangerous agents (humans) via environments, incentives, and “character”; perhaps AI should be shaped toward high‑character, cooperative behavior.
Practical Harms and Misuse Scenarios
- Near‑term worries include: autonomous scam bots, mass persuasion, AI‑boosted addiction, and poorly supervised agents controlling critical infrastructure (e.g., hospitals).
- Some argue existing laws (e.g., against hacking) already cover many harms; others note enforcement gaps and the historical tolerance of significant levels of crime.
Doomerism, Hype, and Lab Incentives
- “Doomer” narratives are viewed by some as marketing aimed at investors and regulators, inflating perceived power while justifying regulatory moats.
- Others think labs at the frontier really are encountering qualitatively new safety issues that smaller players don’t yet see.
- There is broad suspicion that both minimization (“it’s just software”) and maximization (“machine gods”) of risk are shaped by business incentives.