Understand how transformers work by demystifying the math behind them

Original Article ↗ Hacker News Discussion ↗

Transformers as the core architecture behind modern large language models are seen as both deceptively simple in code and overwhelmingly complex in behavior, prompting calls for explanations that bridge intuitive tutorials and rigorous math. Commenters trade resources on mechanistic interpretability, positional encodings, and attention, while correcting misconceptions about backpropagation, neural network terminology, and numerical issues like softmax stability. Alongside technical clarifications, a recurring theme is whether scaling current “next‑token predictors” can ever yield “actual AI,” or if new architectures with learnable compute graphs and stronger reasoning abilities will be needed.

High‑Level Reactions

  • Many find transformers hard to truly “understand”: simple models are interpretable but not useful, while useful models feel too large to grasp.
  • Comparisons to “monad tutorials”: you must wrestle with them, and once you understand them, it becomes hard to explain them clearly.
  • Some appreciate the article; others think the “math behind” framing oversells the level of mathematical depth.

Transformers, Attention, and Compute Graphs

  • One view: the “mystery” is mostly swapping static layer weights for learned Q/K/V matrices and heavy parallelism; attention is seen as structurally simple but rigid.
  • Counterpoint: what’s special is full, lossless access to all previous tokens, unlike RNNs which compress history.
  • Debate over whether future progress requires making the compute graph itself learnable; issues raised about discreteness of graph space and gradient-based training.
  • Suggestions include genetic algorithms, differentiable architecture search, and Hebbian-like mechanisms for linking subgraphs.

Progress, “Actual AI,” and Tool Use

  • Some argue current LLMs plateau around GPT‑3.5/4 and mainly provide efficient “information compression,” not “actual AI.”
  • Others argue functionality has clearly improved and that concept learning and multi-step reasoning (e.g., with ReAct/Tree-of-Thoughts) already go beyond simple lookup.
  • “Actual AI” is variously defined as robust reasoning on unseen problems, autonomous plan construction, or systems that can design complex artifacts end-to-end.
  • Strong support for augmenting LLMs with external tools and simulators rather than trying to emulate CPUs/NAND gates inside the network.

Relation to the Brain and Neural Nets

  • Mixed views on whether the brain “uses transformers”: some say no; others point to work showing transformer-like representations resembling hippocampal grid/place cells.
  • Clarification that transformers are neural networks trained with backpropagation, despite some confusion in the thread about that.
  • Forward‑forward algorithms and other more biologically plausible learning schemes are mentioned as alternative inspirations.

Math, Embeddings, and Positional Encoding

  • Multiple comments request deeper treatment of:
    • Tokenization vs embeddings and “dequantization.”
    • Positional encodings, why sin/cos, frequency choices, and learnable vs fixed encodings.
    • Detailed roles of dot products, softmax, scaling, and residuals in attention.
  • Some praise a one‑page “Transformer in equations” sheet as the kind of concise math they wanted.

Implementation Concerns and Numerical Issues

  • Readers flag possible code issues in the decoder residual connections and missing layer norms.
  • A section attributing NaNs to “gradient explosion” is criticized; the real issue is described as large activations and numerically unstable softmax.
  • There is minor confusion about the positional encoding formulas and indexing; variations from the original paper’s code are noted but thought to be non-critical.

Next‑Token Prediction and Generalization

  • Question: how can “just” next‑token predictors handle unseen identifiers or follow new in-context examples?
  • Answers emphasize:
    • Learned high-dimensional latent concepts (including “gibberish/placeholder”) that new token strings map into.
    • In-context learning and pattern matching over token sequences.
    • Theoretical point: a correct conditional next‑token distribution implicitly defines a distribution over full sequences, capable of representing rich behaviors.