The LLMentalist Effect

Original Article ↗ Hacker News Discussion ↗

Critique of the “LLMentalist / con-artist” analogy

  • Many commenters argue the psychic comparison is overstated: LLMs are continuously checked against ground truth (e.g., running code), while psychics cherry-pick safe claims.
  • Others accept a partial analogy: RLHF can reward answers that sound confident and emotionally satisfying even when unsupported, akin to cold reading.
  • Several think the article is internally inconsistent (both “it’s all an illusion” and “it stole all our work”) and light on technical detail; some see it as motivated by dislike of LLMs and note it’s dated (2023, pre-o1/o3).

What is “intelligence”?

  • Thread repeatedly notes that the article never pins this down.
  • Competing definitions:
    • Narrow/functional: ability to solve problems or use information effectively.
    • Richer: requires awareness, introspection, or “knowing with conscience.”
  • People point out you can’t resolve “are LLMs intelligent?” without first agreeing on a definition; otherwise the debate becomes circular.

Similarities and differences to human cognition

  • Some argue a large part of human cognition is statistical pattern-matching (especially grammar, conversational wandering), so LLMs plausibly mirror an aspect of mind.
  • Others emphasize missing facets: introspection, long-term memory, embodiment, consciousness, non-verbal reasoning, and the ability to notice and correct one’s own failures.
  • A minority worry about “religious” attitudes that insist human thinking must be fundamentally non-mechanical.

Capabilities, generalization, and benchmarks

  • Dispute over whether solving Olympiad/ARC-AGI/logic tasks shows real reasoning or just sophisticated pattern reuse/overfitting.
  • Some highlight LLM weaknesses on basic compositional tasks (like reliably counting letters) to argue limits of next-token prediction.
  • A custom NanoGPT sorting experiment is cited to counter “pure parroting,” sparking a technical subthread on what counts as genuine generalization.

Illusion, ELIZA effect, and world models

  • Several draw parallels to the ELIZA effect: humans over-attribute understanding to fluent text.
  • Others insist LLMs do build internal world models; the “illusion” is partly RLHF pushing them toward persuasive personas.
  • One framing: the chat persona is a fictional character users bond with, not the underlying system.

Usefulness and economics

  • Some say “do they think?” is secondary to “are they useful and cost-effective?”
  • Others see a gap between hype and current utility; examples like Copilot provoke disagreement over whether aggressive promotion reflects real value or a search for one.