Procedural knowledge in pretraining drives reasoning in large language models
Pretraining large language models on step‑by‑step explanations and code appears to teach them reusable procedures for solving problems, rather than merely recalling memorized answers. Commenters link this to the strong performance of code‑heavy training mixes and to new “reasoning” models that can backtrack, while arguing over whether such behavior counts as genuine reasoning or sophisticated pattern extrapolation. The thread also explores implications for training data choices (e.g., textbooks, proofs, student notes) and what current LLM limitations reveal about both machine and human cognition.
Procedural Knowledge vs. Retrieval
- Core claim discussed: LLM reasoning traces on math problems seem driven more by procedural knowledge (step-by-step methods, formulas, code) than by memorized answers to identical questions.
- Commenters emphasize this as evidence of generalization over pure retrieval: models synthesize patterns for “how to solve” rather than just lookup.
- Some note this aligns with experiences that models often follow a reasoning path without self-correction; once on a path, backtracking is weak unless explicitly trained for it.
Memorization, Generalization, and “Reasoning”
- Debate over whether this is “memorization at a higher level” or genuine generalization.
- One view: shared weights force compression into patterns that generalize beyond seen examples.
- Another view: it’s still fundamentally pattern extrapolation, not human-like reasoning.
- Several distinguish “generalization” (pattern-based guessing) from “reasoning” (multi-step, flexible, with alternatives and backtracking), arguing LLMs do some of both but imperfectly.
- Others argue that if models produce correct, novel step-by-step solutions beyond training examples, calling that “reasoning” is justified.
Role of Training Data (Code, Textbooks, Notes)
- Participants link the findings to prior work showing benefits of mixing substantial code into training, especially for tasks needing long-range state tracking.
- Some note that major models use significant code percentages and that mixing text+code can outperform specialized-only training.
- There is interest in training more on textbooks, proofs, student notes, and worked examples, with the idea that procedural content and corrections may especially help reasoning.
- A separate thread connects this to pretraining for chip design, arguing strong pretraining is plausibly necessary for complex design reasoning.
Human vs. LLM Reasoning and Reliability
- Long meta-discussion compares human and LLM fallibility:
- Humans are also unreliable and often on “autopilot,” yet bear responsibility and can be incentivized.
- LLMs are powerful but opaque and hard to hold accountable; complexity, not nondeterminism per se, undermines responsibility.
- Some object to the term “reasoning” as marketing language; others defend everyday anthropomorphic terms (“thinking,” “reasoning”) as convenient approximations.
Impact and Expectations
- Several expect substantial economic and practical impact even from current imperfect models.
- Others stress that hype outpaces reliability, and that LLMs may be best used as a natural-language front-end to more formal tools (code, solvers) rather than as standalone reasoners.