"Superintelligence" 10 years later

What “AI safety” and “safe AI” mean

• One camp equates “safe” with “doesn’t cause human extinction”; others argue that’s necessary but far from sufficient.
• Discussion of “x‑risk” (extinction) vs “s‑risk” (vast suffering). Some fear futures with surviving but subjugated or zoo‑like humans.
• Safety is framed broadly as avoiding catastrophic, dangerous outcomes, but no consensus specification emerges; some note even the field lacks a precise, machine‑checkable definition.

Instrumental convergence & paperclips

• Several comments reference instrumental convergence and the “paperclip maximizer” as canonical worries: powerful optimizers turning the world to some trivial goal.
• Others deride this as a silly, anthropomorphic or “super‑stupid” scenario that has nonetheless captured elite imagination.

Corporations, power, and alignment

• Repeated comparison of corporations to existing “superintelligences” or unaligned optimizers: profit is likened to paperclips.
• Debate whether profit maximization is inherently less harmful than pure paperclip‑style optimization; examples of pollution, climate damage, labor abuse as profit‑seeking side effects.
• Concern that AGI will be built to serve narrow interests (shareholders, governments, militaries), creating extreme inequality or authoritarian control, even if extinction is avoided.

Bostrom’s book and conceptual critiques

• Some see the book as a seminal, accessible introduction that made AI risk mainstream.
• Others find it tedious, overlong, speculative, and mathematically elaborate on ill‑defined concepts (AGI, simulations, “superintelligence”).
• Critiques include: worm‑vs‑human metaphor is misleading; Turing‑machine analogies misuse infinities; “superintelligence” and “omnipotence” debates resemble theology more than engineering.

Current AI reality vs scenarios

• Many note today’s LLMs and generative models are far from the bunker‑born, runaway AGI imagined a decade ago: progress is incremental, competitive, and mostly public.
• Sharp disagreement on whether LLMs are “just autocomplete” vs already the closest thing to AGI; arguments focus on reasoning, planning, learning, and architectural limits.
• Some argue real near‑term risks are mundane: job loss, regulatory arbitrage, opaque decision systems (“weapons of math destruction”), and AI‑driven concentration of power.

Regulation, alarms, and public response

• Analogy to nuclear history: meaningful constraints may only appear after a major AI‑linked disaster; others point to ongoing, diffuse harms that lack a “Hiroshima moment.”
• Skeptical voices frame AI safety rhetoric as fear‑mongering, power‑consolidation, or a replay of past tech panics; others insist on taking both extinction and suffering risks seriously, alongside political and economic dangers.