Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces
Critics of “chain-of-thought” outputs in large language models argue that intermediate tokens like “aha” or “wait, that’s wrong” are being wrongly treated as genuine reasoning or internal thoughts, even though the model is only generating probabilistic text step by step. Commenters warn that this anthropomorphizing can mislead users, researchers, and even policymakers into over-trusting explanations, inferring intent, or ascribing consciousness where none exists, with real-world consequences for safety and regulation. Others note that these traces can still be pragmatically useful for debugging or guiding models, but stress that they should be viewed as tool-generated artifacts rather than transparent windows into machine cognition.
Nature of “thinking traces” and CoT
- Many argue intermediate tokens (“aha”, “wait, that’s wrong”, CoT) are not genuine thoughts or internal state, just next-token predictions conditioned on prior text.
- Some see them as “scratch” or “word vomit” that populates context so attention can retrieve better patterns; others call them rhetorical tricks or “film noir monologues” indistinguishable from character dialogue.
- A minority view suggests there may be a loose but real semantic pattern: e.g., “aha” often precedes a shift toward a better answer, so producing it might nudge later tokens toward useful patterns.
Anthropomorphizing LLMs
- Many posters see anthropomorphizing as a serious problem: it misleads non-experts, fuels “AI psychosis,” and confuses judges and policymakers about what models actually do.
- Others say light anthropomorphism is normal and fun (as with databases or printers) and doesn’t imply real belief in machine minds.
- There’s concern that vendors intentionally choose human-like terms (“reasoning,” “agents,” “companions”) to overstate capability.
- Counterpoint: LLMs display human-like interaction patterns (respond better to encouragement), and ignoring this behavior can be practically irrational, even if it’s just a training-data effect.
Usefulness and risks of exposing reasoning tokens
- Some users find traces useful to catch misunderstandings early on long tasks.
- Others argue users should largely ignore traces; they can be persuasive “sophistry” that makes wrong answers feel trustworthy.
- If traces aren’t faithful to the underlying computation, they’re poor auditing artifacts and easy to misinterpret as true “intent.”
Consciousness, intelligence, and analogy debates
- Extended back-and-forth on whether LLMs “think,” “understand,” or could be “conscious,” with no consensus.
- One side emphasizes mechanistic, fixed-weight systems lacking human-style memory, embodiment, and biological complexity; language ≠ thought.
- Another side argues we lack precise definitions of consciousness and know human introspection is unreliable, so strong claims of non-consciousness are overconfident.
- Analogies are contested: critics compare “LLMs think” to calling lightbulbs “small stars”; others say both bulbs and stars “produce light,” and both humans and LLMs “produce thoughts” in a pragmatic sense.
Terminology, paper style, and criticism
- Several dislike imperative, “clickbaity” titles (“Stop doing X”), seeing them as blog-post style.
- Others note this is explicitly a position paper; strong normative framing is expected.
- Some criticize the paper’s argumentation as internally inconsistent or overly dismissive of distributional semantics.
- There is confusion and debate over what “thinking traces” are if not reasoning; suggestions include “learned prompt augmentation” or purely optimization-driven token patterns.
Engineering and practical implications
- Posters stress building systems that log inputs, model/version, tool calls, and outputs so behavior can be replayed, rather than relying on self-explanations.
- There’s broad agreement that models often reach good conclusions via opaque processes, and that over-interpreting intermediate text as genuine reasoning is unsafe and misleading.