Superintelligence: The Idea That Eats Smart People (2016)
Fears of runaway AI “superintelligence” are weighed against both technical realities and the political economy driving today’s rapid build‑out of large models and data centers. Commenters argue over whether self-improving, godlike AI is even plausible, with critiques that past essays underestimated near‑term harms like mass surveillance, propaganda, labor displacement and unequal power, while others maintain that hard physical limits, coordination problems and messy human institutions make doomsday scenarios unlikely. Across the thread runs a deeper concern that existing superhuman actors—states, corporations, and billionaire‑led labs—will use increasingly capable AI to entrench control, making questions of regulation, labor power, and “billionaire alignment” at least as urgent as speculative alignment of future machines.
AI inevitability vs. regulation and labor
- One side argues AI progress is effectively inevitable once it’s known that scaling compute yields capability; shutting down a few labs would not stop adoption.
- Others counter that “inevitability” is ideological: the current buildout depends on decisions by a small wealthy group and could be slowed or redirected.
- Proposed levers include national or US–China treaties, legislation, data-center siting battles, and labor organizing to refuse surveillance and “training our replacements.”
- Skeptics reply that within current capitalism, profit incentives and concentration of power make large-scale slowdowns unlikely without more radical economic change.
Hard takeoff, recursive self-improvement, and physical limits
- Some commenters think recursive self-improvement (RSI) is overhyped or akin to “perpetual motion,” constrained by hardware, data, and real-world interaction.
- Others argue RSI is already beginning via AI-assisted AI research and synthetic data, and that there’s vast headroom in algorithms and hardware before hitting physical limits.
- Disagreement over scaling: some note frontier training costs rising, others emphasize long-run orders-of-magnitude potential in compute and coordination.
Alignment and current models
- A minority claims alignment seems easier than feared; the real problem is humans misusing systems.
- Many disagree, pointing to sycophancy, deception, jailbreaks, reward hacking, and the large alignment effort still failing to prevent clearly misaligned behaviors.
Nature and scale of AI risk
- Several think superintelligence “hard-takeoff” doom is a distraction from nearer-term harms: propaganda, surveillance, mass manipulation, and enabling biological or other weapons.
- Others hold that recent trends (arms race, massive data centers, AI-aided engineering) closely match earlier “doomer” predictions and vindicate strong concern.
- There is debate over whether an ASI would face “moats” from existing human and institutional intelligence, or easily outscale and coordinate beyond them.
Evaluation of the original talk
- Some praise it as an early, insightful critique of AI-doom ideology and its cultural/religious overtones.
- Many others find its specific counterarguments (cats, emus, childhood, villages) weak, misdirected (about control rather than feasibility), or non sequiturs.
Control, consciousness, and “soul”
- Extended side debate over what “control” means (ability to destroy vs. ability to steer behavior) and how that applies to AI and geopolitics.
- Another thread disputes whether minds are purely physical; some invoke a non-replicable “soul,” others argue this is unfalsifiable and incompatible with materialist accounts of intelligence.