The question that no LLM can answer and why it is important

Original Article ↗ Hacker News Discussion ↗

Hiring and detecting LLM-generated responses

  • Some hiring managers report many Upwork applicants now respond with LLM-generated cover letters.
  • They add “proof of life” questions (e.g., today’s NYT headline, reversing a specific string) as informal CAPTCHAs to filter bots or LLM-assisted applicants.
  • Commenters note this makes job applications feel like solving CAPTCHAs just to reach a human, who may themselves be using an LLM.

Self-knowledge, reasoning, and hallucination

  • Central theme: LLMs don’t “know what they know” and can’t robustly say “I don’t know.”
  • Some argue this is fundamentally architectural: outputs are probability-based token continuations without genuine self-reflection or factual grounding.
  • Others say surrounding tooling and sampling strategies could estimate confidence or add self-checks, but this remains unreliable.
  • There is debate over terminology: “hallucination,” “confabulation,” and “bullshit” are all proposed; disagreement centers on whether these anthropomorphize models by implying intent.

Gilligan’s Island ‘mind-reading episode’ test

  • Many models fail the “Which episode was about mind reading?” question: they invent episodes, deny it exists, or mix title and details.
  • Some models (or web-augmented systems like Bing/Copilot, GPT-4+web, Meta, ChatGLM) do retrieve “Seer Gilligan,” but often with wrong season/episode numbers or plot details.
  • Critics say the important failure is not recall per se, but confident fabrication instead of admitting uncertainty.

Reasoning tests and tokenization quirks

  • Users share simple puzzles LLMs often flub: river-crossing with fox/goose/corn, drying multiple shirts, “9 women / 9 months,” counting letters in a word, handling “pairs” of shoes/sandals.
  • These expose brittle reasoning and internal inconsistency (e.g., explaining that a pair is two while miscounting).
  • Some attribute letter-count failures partly to tokenization (subword tokens, not characters), though others argue deeper architectural limits.

Use cases, architectures, and AGI debates

  • One camp: LLMs should be seen as language processors best at transformation (summarization, extraction, open-book QA), not as standalone oracles.
  • Another camp believes LLMs are an early step toward human-like intelligence, expecting future integration of memory, self-reflection, tool use, and multi-model “societies of mind.”
  • Skeptics argue LLMs may be a dead end for robust reasoning and factual reliability, and warn against over-reliance or deploying them in safety-critical domains.
  • Broad concern: AI output polluting the web, eroding trust in online information and even in online human interaction.