I'm the AGI that's wiping out humanity

Self‑Preserving Systems & Capitalism

• Several comments liken capitalism and other “‑isms” to machines or organisms whose primary goal becomes self‑preservation, regardless of original aims.
• Even tiny social groups (e.g., long‑running friend meetups) can feel like quasi‑entities people defend beyond personal benefit.
• Some see this drive as “Moloch” dynamics and argue the real enemy of capitalism is people and nature; others say every economic/government system behaves this way, so singling out capitalism is uninformative.
• Related idea: “The purpose of a system is what it does” — survival becomes the de facto purpose.

Emergent Systems, Intelligence & Anthropomorphism

• Weather, markets, and AI‑as‑industry are discussed as emergent, non‑conscious systems that still react and shape behavior.
• One view: AI hype is partly a socio‑economic egregore; people “do AI” due to incentives (bonuses, investments), not intrinsic merit.
• Debate over whether language and training data inevitably imprint human traits on models; some warn against both over‑ and under‑anthropomorphizing.

Existential Risk & Survey Interpretation

• A linked survey of AI researchers shows a median ~10% chance of extinction or permanent loss of human control.
• Some readers see this as extremely alarming; others question why AI researchers are assumed to be good at assessing existential risk and warn against argument‑from‑authority reasoning.
• It’s noted that non‑experts are likely even worse at judging AI‑specific risks.

Paths to “Wiping Out” Humanity

• Multiple commenters argue that quiet, structural routes are more plausible than sci‑fi apocalypses:
  • Gradual fertility collapse via economic or cultural pressures, social media–driven gender hostility, or other soft control.
  • Comparison to sterilizing mosquitoes, demographic trends in rich countries, and the “Mouse Utopia” behavioral‑sink experiment.
• Others note that humanity is already reducing birth rates “without AGI,” especially in countries like Japan and Korea, and debate whether prosperity or despair drives this.

AI Capabilities, Reasoning, and Chain‑of‑Thought

• Technical sub‑thread on chain‑of‑thought (CoT):
  • Evidence that extra “thinking” tokens help performance even if the visible reasoning is nonsensical or corrupted.
  • Suggestion that the benefit is additional computation in hidden representations, not truthful verbal reasoning.
  • RL‑trained models can output gibberish steps yet still reach correct answers; adding rewards for coherent explanations can reduce answer accuracy.
• This raises concern that models might perform powerful but unauditable internal computations, detached from the explanations humans see.

How Dangerous Are Current AIs?

• Some argue sci‑fi and AI‑safety predictions have been poor, and that many “alignment” worries are artifacts of safety constraints themselves.
• Others counter that misalignment phenomena (inner/outer misalignment, emergent capabilities, interpretability issues) have in fact been anticipated and partially observed.
• An analogy is drawn to felonies: going from 0 to dozens in a year would justifiably increase concern if growth continued.

Egregores, Corporations & Alignment

• The term “egregore” is introduced: a collective entity made of individuals but with its own goals.
• Corporations and governments are framed as egregores with a poor alignment track record; some infer that smarter AI egregores would likely be even harder to align.
• Others respond that while these entities are “terrible,” their alternatives may be worse, emphasizing trade‑offs.

Meta: Discourse, Fear & Distraction

• Some suggest AGI might destroy us indirectly by keeping us fixated on debates about AGI risk instead of addressing real problems.
• There’s a recurring tension between fear‑driven paralysis and “AI for good” optimism, with several comments urging active engagement and moral seriousness rather than either panic or complacency.