Transformers Can Do Arithmetic with the Right Embeddings
Researchers claim that transformers can reliably perform long-digit arithmetic if numbers are embedded with explicit positional structure (e.g., least significant digit first), highlighting how current tokenization and formatting choices hinder mathematical ability. Commenters are split on the significance: some see it as a useful probe of transformers’ capacity for algorithmic reasoning and a step toward better "world models," while others dismiss it as a narrow hack that doesn’t address broader reasoning limits. The thread branches into a wider debate over whether current LLM architectures can truly generalize and reason, or whether they will always need external tools (like calculators) and specialized inductive biases for tasks beyond pattern matching.
Arithmetic & Representations in Transformers
- Several comments argue LLMs struggle with math because of how numbers and positions are encoded, not because transformers cannot learn arithmetic.
- Lack of “column” structure and left‑to‑right output conflicts with right‑to‑left digit operations; this makes addition effectively harder (more than linear) for a vanilla transformer.
- Thread notes that common tokenizers split numbers into odd chunks (e.g., “12345678” → “123”, “456”, “78”), obscuring digit positions and magnitudes.
- Some suggest reversing numbers (least significant digit first) and adding position-aware embeddings; this is exactly what the paper and prior work explore.
What This Paper Adds (According to the Thread)
- Introduces specialized positional/column embeddings for digits, effectively “number-flipping” and aligning digits by significance.
- Shows sharp improvements on multi‑digit addition and some transfer to related tasks like sorting and limited multiplication.
- Several commenters see it as evidence that better encodings remove key barriers and reveal transformers’ “logical extrapolation” abilities.
Skepticism and Limits
- Critics see this as a narrow, hand‑engineered hack: strong inductive bias tailored to addition, with weak or no gains for subtraction/division.
- Many stress 99% accuracy on 100‑digit arithmetic is useless for calculators or safety‑critical domains; real systems need external exact tools.
- Some argue it doesn’t demonstrate genuine algorithm learning, just embedding design that partially bakes in the structure of addition.
Reasoning vs Pattern Matching
- Ongoing debate: are transformers “reasoning” or just high‑dimensional curve fitters?
- Pro‑side: success on controlled arithmetic tasks and broader benchmarks suggests nontrivial reasoning, albeit imperfect and fragile.
- Skeptical side: failure to generalize from small to arbitrary‑length arithmetic and brittleness on reasoning benchmarks indicate poor, highly domain‑bound reasoning.
Broader Architectural & Evaluation Ideas
- Several propose hybrids: LLM cores plus embedded ALU / CAS / search tools, perhaps via special tokens rather than plain text calls.
- Others emphasize richer, task‑aware embeddings as the new “feature engineering” frontier.
- Discussion touches on AGI definitions: calls for percentile‑based, task‑wise benchmarks instead of a binary “has AGI / hasn’t” framing.