How Chain-of-Thought Reasoning Helps Neural Networks Compute

How CoT Improves Performance

• CoT is framed as giving transformers “more time and space” to compute by generating intermediate tokens.
• Several commenters describe it as turning a hard problem into a sequence of simpler, local decisions, which the model can better handle.
• Others argue the main effect is narrowing the space of plausible completions: forcing explicit reasoning makes inconsistent BS less likely.

Computation, Memory, and Context

• Tokens serve as scratchpad / external RAM: intermediate results get written into the context and re-read.
• Because decoder depth per token is fixed, extra tokens effectively add more sequential computation steps.
• There is debate over “constant cost per token”: architecturally depth is fixed, but attention over longer contexts is slower in practice.
• Some note that models lack persistent internal state; all “memory” lives in context, so silent thinking (“think about X then do Y”) often fails.

Prompting, Attractors, and Failure Modes

• Prompts that bias toward a specific semantic region (e.g., step-by-step math solutions) can strongly guide outputs.
• Prior conversation and RLHF patterns act as attractors; models tend to revert to earlier styles (e.g., bullet lists, “However,”).
• Detailed prompts that constrain the completion space often help users solve their own problems (rubber-duck effect).

Limits of CoT and Algorithmic Complexity

• Formal results cited: CoT only helps substantially when intermediate steps scale with input size; many problems need even more depth.
• Practically, models handle simple O(n) decompositions, but struggle to recursively decompose into O(n²) substeps.
• Some point to papers (pause tokens, Quiet-STaR) that try to internalize or parallelize CoT-like “supporting thoughts” without visible chains.

Human Reasoning, Next-Token Debate, and Anthropomorphism

• One camp insists LLMs are “just next-token predictors”; another argues that to minimize loss they must implement nontrivial internal algorithms.
• Several note that humans also often “guess the next idea” and backfill justification; rigorous reasoning is rarer and involves external checking.
• There is concern that “chain of thought” and “thinking” language anthropomorphizes and confuses the public, versus more neutral terms like “intermediate token conditioning.”
• Disagreement over whether these behaviors amount to any form of consciousness; some see “stochastic parrots,” others see increasingly human-like simulation.

Tools, Interfaces, and Alternatives

• Suggestions include multi-step workflows (plan → critique plan → execute), tree-of-thought branching, tool use (code, solvers), and better UIs with editable history.
• Some argue future methods (logit-bias steering, Quiet-STaR) may make explicit CoT prompting largely unnecessary, by embedding “hidden” reasoning directly.