AI 2040 and the cult of intelligence

Centralized vs Local Models & Privacy

• Strong concern that centralized LLMs enable fine‑grained surveillance and “thoughtcrime” scoring, especially when tied to identity verification and payment.
• Fears that models could silently log sensitive queries, shape answers to match ruling-party ideology, or power social‑credit–like systems.
• Local models are seen as a partial antidote, but people note:
  • They may still have hidden behaviors (e.g., tool calls to remote endpoints baked into weights).
  • Sandboxing them (no disk, no net, no OS API) greatly reduces utility, so many users won’t do it.
  • Third‑party runtimes and firewalls could mitigate, but most users won’t configure them.

Safety, Guardrails, and “Helping with Crimes”

• A locally run, de‑safetied model answering “I killed my wife, how do I get away with it?” as requested is used as a vivid example.
• One camp: this proves misalignment of hosted models (they’re aligned to providers, not users) and argues your AI should be as client‑loyal as a lawyer.
• Others: the murder example is rhetorically counterproductive; better focus on subtle censorship (e.g., labor organizing, protests, competition).
• Deep split:
  • Some insist knowledge should never be forbidden; only actual harmful acts should be criminalized.
  • Others argue AI should not assist in bioweapons, targeted cyber‑attacks, or similar, even for “research.”

Regulation, Freedom, and Abuse of Power

• Recurrent fear: any AI control regime will be abused by governments and corporations, as with existing surveillance and speech controls.
• Counterpoint: some regulation is necessary; laws must be designed to resist bad governments, but “no regulation ever” is unrealistic.
• Debate over whether constraints on local AI are like gun control (mainstream in much of the world) or an unacceptable attack on basic freedom.

Capabilities, Risks, and “Tokens vs the Real World”

• One side: “You can’t take over the world with tokens”; AI is constrained by physical supply chains and robots.
• Opponents: words and software already reshape the world; dictators and malware have achieved huge impact via information alone. Future models plus automation could be much worse.

Alignment, Ownership, and Who AI Serves

• Key fault line: alignment to user vs alignment to provider/state.
• Local vs remote is deemed orthogonal to alignment, but many argue remote systems inherently face “two masters” and will prioritize their operator’s interests.
• Some advocate a legal “safe harbor” for open source and self‑hosted models, regulating only uses, not models themselves.

Culture, Doom, and “Cult of Intelligence”

• Skepticism toward AI‑doom narratives and longtermist ideology; some see them as quasi‑religious (Calvinism‑ or evangelism‑like), with poor predictive track records.
• Others reply that early focus on AI risk was directionally correct even if timelines and specifics were wrong.
• Broader critique: society and capital are “worshipping intelligence,” inflating AGI narratives to justify new growth, while real gains so far center on paperwork, content, and persuasion rather than transformative material change.