Pretraining data enables narrow selection capabilities in transformer models

Scope and Methods of the Paper

• Paper studies transformer generalization on synthetic sinusoidal functions using a small GPT‑2–scale model (~9.5M params).
• Many commenters argue this is a highly constrained, toy setup and may not transfer cleanly to natural language tasks.
• Some note more recent/meta-learning work (on task diversity and in‑context learning) may supersede or contextualize these results.
• Criticism: media and social media amplify this paper because it supports a preferred “LLMs hit a wall” narrative, despite being a narrow result.

Interpolation vs Extrapolation, and “Generalization”

• Core claim (as interpreted by commenters): transformers are near‑optimal at interpolation within the training distribution, but perform poorly on out‑of-domain extrapolation.
• Several participants say this is expected of deep learning in general and not unique to transformers.
• Others question whether “pure extrapolation” is meaningful or even something humans reliably do; much human reasoning may also be sophisticated interpolation.
• Some argue that in high dimensions “everything is extrapolation,” challenging the interpolation/extrapolation dichotomy.

Real-World Behavior of LLMs

• Multiple anecdotes: GPT‑4 can handle novel APIs, internal UI frameworks, hypothetical or “franken-” languages, and other unseen structures with few examples.
• This is cited as evidence of at least some real generalization, even if bounded and imperfect.
• Counterpoint: these may still be within a very broad training distribution (“supercharged interpolation”) due to massive web-scale data.

Mathematical and Algorithmic Limits

• Long subthread on why LLMs struggle with arithmetic.
• Explanations offered:
  • Models approximate token distributions, not explicit algorithms.
  • Tokenization of numbers (e.g., multi-digit tokens) makes learning systematic rules hard.
  • Larger or specialized models and structured numeric tokenization can learn multi-digit operations, but not perfectly.
• Disagreement over whether LLMs “learn algorithms” or only approximate them; some cite experiments and grokking-like behavior as evidence of emergent algorithmic reasoning.

Broader Implications for AGI and Human Comparison

• Some connect extrapolation limits to skepticism about runaway “AI self-improvement.”
• Others argue there is no clear logical barrier to AI designing better AIs; the question is timing and economics, not principle.
• Several note that most humans also do not push knowledge frontiers and mostly reuse existing information, similar to LLMs trained on large text corpora.