Every model learned by gradient descent is approximately a kernel machine (2020)

Original Article ↗ Hacker News Discussion ↗

Gradient-descent–trained neural networks can be mathematically recast as kernel machines, but commenters argue this equivalence is mostly of theoretical interest: the implied kernels depend on the entire training trajectory and don’t offer the practical benefits of classic kernel methods. The thread branches into a broader debate over whether modern deep models—especially large language and generative video models—are primarily memorizing training data or genuinely generalizing, touching on efficiency, the role of associative memory, and what these behaviors imply for claims about artificial general intelligence.

Kernel machines vs neural networks

  • Several commenters note that simple neural networks have long been known to relate to kernel methods and Gaussian processes; the 2020 paper is seen by some as more expository than original.
  • A key criticism: the constructed “kernel” depends on the full training trajectory of gradient descent. This makes it unlike classical kernel methods, where the kernel is mostly data-independent (e.g., Gaussian with tunable parameters).
  • Because every new data point changes the entire training path, you don’t get kernel-style incremental updates; all the complexity may just be “moved into the kernel,” limiting practical insight.
  • Others point out that many ML methods are, in a loose sense, sophisticated memorization over a feature representation, so calling gradient-descent-trained models “kernel machines” may be technically true but not very illuminating.

Memorization vs generalization in modern models

  • A major subthread debates whether striking results from large models (LLMs, video models) are mostly memorization or evidence of broader generalization.
  • One side argues: fine-grained visual or textual detail likely comes from regurgitating training data; opaque train/test splits enable “data laundering.”
  • The other side counters with tasks that appear to require genuine pattern learning and algorithm induction (e.g., in-context regression, grammar-following, Go-playing systems, novel image compositions).
  • Disagreement centers on the definition of “memorization”: exact reuse of examples vs learning compressed, high-level representations and algorithms from examples.

Human intelligence, memory, and AI comparison

  • Many comments draw parallels: human experts (plumbers, lawyers, engineers, chess grandmasters) rely heavily on memory and pattern reuse; creativity is often derivative.
  • Others push back, emphasizing human capacity to handle genuinely novel situations, reuse skills across domains, and explain reasoning—capabilities current models only partially match.

Limits, failures, and confabulation

  • Image examples (e.g., a horse in a space suit) illustrate both creativity and “basic” inconsistencies (helmet or suit not fully enclosing the body), raising doubts about whether such confabulation can be fully eliminated.
  • Some see these as minor artifacts compared to the overall sophistication; others treat them as evidence of missing world-model or logical structure.

Meta: reception of the paper and broader culture

  • Some see the “everything is a kernel machine” framing as overstated or even “nonsense” in the practical sense, given the data-dependent kernel.
  • There is mention of ongoing tension between kernel/SVM-style approaches, classical statistics, and deep learning communities, with some researchers strongly disliking deep learning’s dominance.