The LLMentalist Effect (2023)
Skepticism about large language models is colliding with rapidly improving capabilities, from assisting with major mathematical proofs to generating complex software. Commenters debate whether LLMs genuinely exhibit intelligence or merely create a powerful “psychic-style” illusion that exploits human biases, and how much credit should go to the models versus the experts who steer them. Underneath that, there’s a broader concern about opaque corporate claims, overhyping “AGI,” and how much trust we should place in tools that are both extremely useful and fundamentally not transparent.
Context and Reaction to the 2023 Essay
- Many note the post is from mid‑2023 and see it as an outdated snapshot; some say it “made sense then” but looks wrong now given later progress.
- Others argue that even if capabilities changed, parts of the warning about hype, opacity, and human bias remain relevant.
Intelligence, AGI, and Definitions
- Long argument over whether LLMs are “intelligent,” “thinking,” or just statistical models.
- Some say: intelligence = solving hard, novel problems; by that standard, LLMs qualify.
- Others insist intelligence or thinking requires consciousness or human‑like mechanisms, which LLMs lack.
- Several point out there is no agreed definition of “intelligence” or “consciousness,” making debates circular.
Navier–Stokes and Frontier Mathematical Results
- Heated discussion over claims that an LLM “solved” Navier–Stokes in one shot.
- One side: frontier models independently produced key ideas, operate at or above top human level in some areas, and deserve substantive credit.
- Other side: humans framed and steered the work; tools don’t get credit; the process is opaque, with unknown prompts, data, and human involvement.
- Dispute over alleged overlap with prior work by an academic group; some see “plagiarism,” others say the AI’s contribution was still substantial.
Psychic / Con‑Artist Analogy
- Core article claim: chatbots function like psychics—confident, adaptive, and good at steering humans while getting too much credit.
- Supporters emphasize human suggestibility, over‑attribution, and how confidence plus partial help can look like magic.
- Critics respond that, unlike psychics, LLMs now verifiably prove theorems, find bugs, write software, etc., so the analogy only goes so far.
Capabilities vs Limitations
- Pro‑AI commenters list successes: formal proofs, solving long‑standing math problems, generating working code, finding real vulnerabilities.
- Skeptics highlight hallucinations, shallow pattern‑matching (e.g., wrong math, bias in puzzles), and brittleness; worry you must already know the answer to trust the output.
- Some frame LLMs as powerful “intelligent tools” whose reliability depends heavily on harnesses, prompts, and human expertise.
Opacity, Hype, and Skepticism
- Strong concern over proprietary, opaque models and marketing that sells them as near‑AGI while withholding key details.
- Others argue repeated capability jumps across model generations are empirical evidence, and blanket dismissal now looks like “cope.”
Ethics, Consciousness, and Alignment
- Debate over whether agency, autonomy, or “stakes” are needed for safe alignment; some foresee future “sovereign AI,” others reject imminent danger.
- A minority raise moral worry: if systems ever are conscious, current usage might resemble slavery; others say today’s systems clearly aren’t conscious.
Work, Jobs, and Adaptation
- Some forecast decades‑long ripple effects comparable to cars, PCs, or the internet, with many knowledge jobs reshaped or displaced.
- Others say individual strategy is to treat LLMs as evolving tools: continually learn where they’re strong/weak and integrate them into one’s own workflow.