Hallucination is inevitable: An innate limitation of large language models

Original Article ↗ Hacker News Discussion ↗

A recent paper arguing that hallucinations are an unavoidable consequence of how large language models compute answers has triggered broader scrutiny of what “hallucination” even means and whether it’s meaningfully different from ordinary inaccuracy. Commenters contrast LLM behavior with human reasoning, debating whether these models have any genuine “world model” or self-knowledge of when they don’t know, and whether additional supervisory layers or confidence estimators could curb false but fluent outputs. Many conclude that perfect reliability is unrealistic; the real question is how to bound, detect, or even productively harness hallucinations rather than eliminate them entirely.

What “hallucination” means and whether the term fits

  • Many argue the paper’s definition (“inconsistencies with a computable ground‑truth function”) is just “being wrong” or fabricating, not true hallucination.
  • Several commenters dislike the word, saying it anthropomorphizes LLMs and makes failure sound mystical or impressive; “confabulation” or “bullshitting” are suggested as better terms.
  • Others note humans also construct internally consistent but false stories, so “hallucination” or “confabulation” has some intuitive appeal.

Theoretical inevitability vs practical importance

  • The paper’s argument is seen as a diagonalization / complexity result: bounded models (e.g., polynomial‑time LLMs) cannot compute all functions, especially some NP‑hard ones, so there will be inputs where they fail.
  • Critics say: this is true of any finite system (including brains) and doesn’t by itself say how often or how badly failures occur in realistic domains.
  • Some point out the result explicitly disallows “I don’t know” and is about adversarially constructed truth functions, so it may say little about the everyday “hallucination problem”.

LLMs, world models, and understanding

  • One camp: LLMs are just next‑token predictors with no world model, concepts, or self; coherent language is a statistical illusion.
  • Opposing camp: internal structure (e.g., chess/Othello board representations) shows learned world models in the weights, even if static and opaque.
  • Long subthread debates whether indistinguishably human‑like behavior is sufficient to call this “understanding”, and whether that notion even has a crisp definition.

“I don’t know”, confidence, and self‑reflection

  • Many see better uncertainty handling as the real problem: getting models to say “I don’t know” or qualify answers instead of guessing.
  • Suggested mechanisms:
    • External control loops (RAG, search, tools, secondary “safety” or evaluator models, majority vote across prompts).
    • Internal signals (perplexity, volatility across slightly perturbed prompts, confidence scores) to gate or down‑scope answers.
  • Skeptics argue current transformers lack genuine self‑reflection; any confidence score is itself just another token sequence.

Comparisons to human cognition

  • Humans also misremember, confabulate, follow group narratives, and reason poorly; some see LLM “hallucinations” as analogous, others emphasize humans’ better meta‑cognition and ability to withhold answers.
  • Several note that hallucination‑like generativity is also where creativity and idea generation come from; the issue is not hallucination per se but distinguishing when it’s acceptable.

Mitigations, system design, and hype

  • Proposed mitigations: modular architectures (LLM + symbolic logic / knowledge bases), supervisory modules akin to brain “executives”, open‑world recognition, and stronger evaluation pipelines.
  • Some argue LLMs will remain just one component in larger AGI‑like systems rather than “god models”.
  • Thread contains both optimism about continued scaling and architectural advances, and strong skepticism that LLMs alone can reach reliable AGI or hallucination‑free behavior.