"Superintelligence" 10 years later
Ten years after Nick Bostrom’s *Superintelligence*, commenters are sharply divided over whether existential AI risk is a serious, near-term concern or an overblown, speculative thought experiment. Some argue that aligning a superintelligent system with all of humanity’s values is extraordinarily hard and that even non‑extinction outcomes (such as extreme suffering or entrenched techno-authoritarianism) warrant urgent work on “AI safety.” Others counter that current systems like LLMs are far from AGI, that real harms today look more like corporate power concentration, labor displacement, environmental cost, and opaque automated decision-making, and that fears of paperclip-style apocalypses distract from these concrete risks.
What “AI safety” and “safe AI” mean
- One camp equates “safe” with “doesn’t cause human extinction”; others argue that’s necessary but far from sufficient.
- Discussion of “x‑risk” (extinction) vs “s‑risk” (vast suffering). Some fear futures with surviving but subjugated or zoo‑like humans.
- Safety is framed broadly as avoiding catastrophic, dangerous outcomes, but no consensus specification emerges; some note even the field lacks a precise, machine‑checkable definition.
Instrumental convergence & paperclips
- Several comments reference instrumental convergence and the “paperclip maximizer” as canonical worries: powerful optimizers turning the world to some trivial goal.
- Others deride this as a silly, anthropomorphic or “super‑stupid” scenario that has nonetheless captured elite imagination.
Corporations, power, and alignment
- Repeated comparison of corporations to existing “superintelligences” or unaligned optimizers: profit is likened to paperclips.
- Debate whether profit maximization is inherently less harmful than pure paperclip‑style optimization; examples of pollution, climate damage, labor abuse as profit‑seeking side effects.
- Concern that AGI will be built to serve narrow interests (shareholders, governments, militaries), creating extreme inequality or authoritarian control, even if extinction is avoided.
Bostrom’s book and conceptual critiques
- Some see the book as a seminal, accessible introduction that made AI risk mainstream.
- Others find it tedious, overlong, speculative, and mathematically elaborate on ill‑defined concepts (AGI, simulations, “superintelligence”).
- Critiques include: worm‑vs‑human metaphor is misleading; Turing‑machine analogies misuse infinities; “superintelligence” and “omnipotence” debates resemble theology more than engineering.
Current AI reality vs scenarios
- Many note today’s LLMs and generative models are far from the bunker‑born, runaway AGI imagined a decade ago: progress is incremental, competitive, and mostly public.
- Sharp disagreement on whether LLMs are “just autocomplete” vs already the closest thing to AGI; arguments focus on reasoning, planning, learning, and architectural limits.
- Some argue real near‑term risks are mundane: job loss, regulatory arbitrage, opaque decision systems (“weapons of math destruction”), and AI‑driven concentration of power.
Regulation, alarms, and public response
- Analogy to nuclear history: meaningful constraints may only appear after a major AI‑linked disaster; others point to ongoing, diffuse harms that lack a “Hiroshima moment.”
- Skeptical voices frame AI safety rhetoric as fear‑mongering, power‑consolidation, or a replay of past tech panics; others insist on taking both extinction and suffering risks seriously, alongside political and economic dangers.