Why Are LLMs So Gullible?

Original Article ↗ Hacker News Discussion ↗

Claims that large language models are “gullible” prompt debate over whether their failures reflect child‑like intelligence, simple statistical pattern matching, or misaligned incentives to always be helpful. Commenters argue over how far human metaphors should be taken, with some stressing that LLMs lack consciousness, goals, and genuine understanding, while others note that their behavior increasingly resembles human reasoning despite being trained only to predict text. The thread also touches on security and safety implications, such as jailbreaks and prompt injection, and whether future systems should be more distrustful of user input to avoid harmful behavior.

Anthropomorphism and “child-level” comparisons

  • Some argue LLMs are at a “child level” of development and will progress through analogous stages.
  • Others push back: human cognitive development is poorly defined, and mapping LLM progress onto child stages is seen as misleading or speculative.
  • Counterpoint: using human developmental tables as rough capability benchmarks is defended as pragmatically useful, even if it doesn’t imply LLMs are literally like children.

Do LLMs actually reason, or just autocomplete?

  • One side stresses LLMs as “advanced autocomplete” trained only for next-token prediction, not genuine understanding or abstract reasoning.
  • Others argue this dismisses emergent behavior: even simple mechanisms can yield complex reasoning-like abilities, and we lack a precise definition of “thinking” to rule this out.
  • There is tension between viewing LLMs as “stochastic parrots” vs. potentially capable function approximators that might approximate human-like thinking given scale and architecture.

Gullibility, alignment, and trust

  • Some frame gullibility as a byproduct of models being heavily trained to follow user instructions (RLHF), so they “go along” even with adversarial prompts (e.g., “napalm grandma”).
  • Debate over whether we’d actually want future systems to distrust or override users; a more “distrustful” model might be safer but also less usable.
  • Others object that attributing “trust,” “values,” or “honesty” to current LLMs is anthropomorphic; these are just biases in pattern generation, not inner motivations.

Statistical optimization vs human cognition

  • One view: statistical optimization is well-specified and mechanistic, whereas cognitive reasoning is not, making direct comparison suspect.
  • Others counter that we also don’t fully understand trained networks’ internal “steps,” so the contrast is overstated.
  • There’s disagreement over whether human brains are fundamentally different from, or just more efficient variants of, large statistical pattern learners.

Safety, filters, and jailbreak defenses

  • Suggestions include using fixed “invisible prompts” or separate AI filters to pre- and post-process inputs/outputs.
  • Critics note that what counts as “offensive” or “straightforward” is itself fuzzy and context-dependent, making perfect filtering hard.
  • Jailbreaks are seen as exploiting out-of-distribution, non-sequitur contexts where safety conditioning is weak.

Meta-level disagreements

  • Some complain the thread is full of strong but weakly supported claims on both sides.
  • A recurring theme: we lack clear, testable definitions of intelligence, reasoning, or understanding, so confident declarations about what LLMs can “never” be or do are viewed as premature.