The avalanche that hit math is coming for AI development

Claims that AI systems will soon recursively accelerate their own development and rapidly surpass human experts draw sharply mixed reactions here. Commenters weigh anecdotes about powerful research agents and automated math breakthroughs against everyday experiences of buggy code assistants, arguing over whether progress follows an exponential curve or will plateau. Underneath the technical debate are worries about concentration of power, misaligned incentives and regulation, and whether current AI hype meaningfully maps to real-world productivity or existential risk.

Perceived Hype vs Reality

  • Several commenters view the piece as overhyped, built on shaky analogies, speculative graphs, and probabilities “from thin air.”
  • Others report daily experience with current AI tools: good at code and constrained technical tasks but requiring heavy prompting, harnesses, static tooling, and still producing many errors.
  • There is concern about an “avalanche of slop” in AI research and publishing, now mass‑produced rather than just sporadically low‑quality.
  • Some argue LLMs are fundamentally best at writing code and handling verifiable, formal domains (e.g., Lean proofs), not broad “understanding.”

Capabilities, RSI, and Benchmarks

  • Supporters cite internal lab anecdotes: agents that rewrite long‑standing models in days, auto‑diagnose experiments, and run large ablation sweeps with minimal human steering.
  • Others stress that truly transformative recursive self‑improvement would require progress on non‑verifiable tasks, which are hard to benchmark without human judgment.
  • A counterpoint is that there are already many verifiable sub‑tasks in AI R&D (performance tuning, orchestration, code generation) that can drive a self‑reinforcing loop even if “messy” tasks lag.

Economic and Power Dynamics

  • Some fear AI will wipe out both white‑ and manual‑labor jobs, enabling a tiny elite to hoard wealth and AI capability while pacifying others (UBI + VR). Others call UBI politically impossible and see continuity with “old ways.”
  • One line of argument: if superintelligence truly made money obsolete, rational actors would hoard raw materials, land, energy—not cash—so current money‑seeking behavior implies insiders don’t really believe near‑term ASI.
  • Others respond that competitive pressure forces labs to pour everything into compute to “win the race,” regardless of longer‑term resource logic.

Governance and Risk Debates

  • Proposed futures range from pausing AI, to cautiously slowing it, to allowing rapid progress and hoping alignment works, to extreme global surveillance to prevent further development.
  • Feasibility of strict control is questioned, given that ML can run on relatively modest hardware.
  • Some see ASI‑driven takeover scenarios (cyberattacks, military disruption, nanotech) as plausible; others regard them as effectively extinction scenarios that no rational actor seeking profit would risk.

Terminology, Polarization, and Psychology

  • There is dispute about “superintelligence” vs “AGI,” especially after political attempts to rebrand AI; some think using the term now looks bad, others insist it long predates politics.
  • Commenters describe two (or three) psychological camps: rapid‑progress believers, skeptics who continually move goalposts, and those criticizing anyone who attaches concrete dates like “2030” while claiming unpredictability.