LLMs cannot find reasoning errors, but can correct them

Original Article ↗ Hacker News Discussion ↗

Large language models often fail to detect their own reasoning errors but can frequently “fix” answers when told a mistake exists or where it lies. Commenters contrast this behavior with true logical reasoning, debate whether models are merely pattern-matching versus thinking, and explore tactics like multiple drafts, model ensembles, and prompt engineering to improve reliability. The exchange highlights both the impressive surface capabilities of current LLMs and their persistent weaknesses in robust, self-directed error checking and formal reasoning.

Scope of the Paper’s Claim

  • Thread agrees the paper shows: current LLMs often fail to locate their own reasoning errors, but can often improve answers when:
    • They are told that an error exists, and
    • They are given (or strongly hinted at) the error’s location.
  • Several argue this is closer to “trying again with a constraint” than genuine self-correction or reasoning.

Do LLMs Actually Reason?

  • Many commenters insist LLMs don’t truly “reason”; they pattern-match against training data and generate text that resembles reasoning.
  • Others note that, behaviorally, LLMs can follow chains of logic in many examples and even do stream-of-thought, so the line between “pattern-matching” and “reasoning” is fuzzy.
  • Strong skepticism: LLMs can’t consistently maintain logical consistency across multiple dependent clauses or math steps; when corrected, they often just fail in new ways.
  • There is disagreement on definitions of “reasoning”; some want formal, logic-based definitions, others allow more heuristic, human-like notions.

Error Detection vs Error Correction

  • Key distinction: detecting that an answer is wrong versus producing a better alternative once nudged.
  • Several note LLMs are easily persuaded to change even correct answers (“lack of conviction”), largely attributed to alignment/RLHF and a tendency to please the user.
  • Observations that LLMs sometimes correct themselves when told only “this is wrong” without details, but they’ll also “fix” non-errors if asked.

Self-Critique, Perspective, and Training Data

  • Hypothesis: models may critique “someone else’s” answer better than “their own,” because training data has far more examples of people correcting others than themselves.
  • Suggested trick: present the model’s prior output as if written by another person to boost error-finding.
  • Debate on whether this is anthropomorphizing or simply exploiting patterns in highly human-like training data.

Multi-Model and Multi-Sample Approaches

  • Ideas discussed:
    • Using one model to evaluate or attack another’s output.
    • Running several generations (n samples) and picking the best with a smaller selector model.
    • GAN-like setups where a “validator” model is trained explicitly to find errors.
  • Benefits: better error rates and “second chances.”
  • Costs/constraints: higher compute, possible incompatibility with streaming interfaces, still doesn’t solve fundamental reasoning limits.

Prompting, Style, and Practical Use

  • Higher-quality, explicit prompts (good grammar, clear constraints, “think carefully” instructions) noticeably improve results.
  • Some use separate LLM passes to rewrite messy prompts into cleaner, more “canonical” inputs before asking the main model.
  • Observed “context poisoning”: long interactions accumulate hallucinations; restarting with “here’s some bad code, fix it” often yields better corrections.

Interpretability and Understanding

  • Agreement that we understand the training algorithm and code, but not the detailed internal representation linking weights to concepts and reasoning-like behavior.
  • Analogy: we know how to run the optimization, but not how the resulting massive parameter set “implements” specific cognitive-like abilities.