How do neural networks learn?

Original Article ↗ Hacker News Discussion ↗

Researchers are proposing a new mathematical tool, the “neural feature matrix,” to describe how deep neural networks identify and weight important features in data, with claims it can explain phenomena like pruning, grokking, and emergent capabilities and improve performance on tabular tasks. Commenters contrast this targeted, feature-level view with the simplistic “it’s just gradient descent” explanation, arguing that while we know the training mechanics, we still lack a satisfying theory of what complex models like transformers actually learn and how to reliably predict or control their behavior. The exchange highlights a broader tension between incremental mathematical insights, practical engineering progress, and deeper questions about interpretability, generalization, and even what it means for a system to “understand.”

Reading/access and paywalls

  • Many find the original site nearly unusable on mobile due to heavy, intrusive ads.
  • Various ad-blocking and privacy tools (DNS-level blockers, browsers, reader modes) are discussed as workarounds.
  • Some note the irony of “democratizing AI” while the referenced paper is paywalled; others link free arXiv versions.

Scope and substance of the paper

  • Several suspect it might be another over-simplified “physics explains NNs” paper that only applies to idealized feed-forward nets.
  • Others argue the work is narrower and more concrete: characterizing how deep networks develop “features” and how this relates to gradients and training dynamics.
  • One summary highlights claimed explanations for pruning, grokking, and an associated kernel-based method that performs well on tabular benchmarks.
  • There is mild surprise that the core quantity (outer product of weights) could be novel.

What it means to “understand how NNs work”

  • Participants distinguish between:
    • Knowing the training algorithm (gradient descent) vs. understanding internal representations and behavior.
    • “How they work” (mechanism) vs. “what they learn” (features, circuits, abstractions).
  • Proposed criteria for real understanding include:
    • Predicting which architectures/hyperparameters will work on given data.
    • Explaining initialization, weight changes from specific samples, and how to fix particular errors.
    • Providing prescriptive tools, not just descriptive stories.

Debates on understanding, explanation, and philosophy

  • Long subthread on what “understanding” even means, invoking philosophical debates and whether poorly defined questions can still be fruitful.
  • Some see “it’s just gradient descent / curve fitting” as trivially true but practically uninformative, akin to saying software is “just logic gates.”
  • Others defend the reductionist view: NNs are error-minimizing function approximators; the mystery is overstated.

Emergent behavior and generalization

  • Discussion of “emergent properties”: unexpected capabilities (e.g., translation) that appear when scaling models and data.
  • Disagreement over how much these depend on initialization/order of training vs. largely on scale and data.
  • Generalization, brittleness to data shifts, and the inability to formally verify behavior are highlighted as core open issues.

Neural Feature Matrix (NFM) as a tool

  • The NFM (WᵀW per layer) is viewed as:
    • Proportional to average gradient outer products over training data.
    • A potential metric on input space reflecting which directions the network “cares about.”
  • Some see the proportionality as almost obvious; others emphasize the value is in what the NFM enables:
    • Tracking feature importance over training.
    • Explaining pruning and grokking patterns.
    • Driving a kernel method that iteratively updates its kernel via gradient statistics.
  • There is debate over whether a static, global metric like WᵀW can faithfully reflect curved data manifolds, or whether it’s still a useful, interpretable approximation.

Interpretability, safety, and engineering vs theory

  • Several note we “know” the math and code, yet lack insight into specific learned models (especially large transformers).
  • Some frame modern NN work as largely empirical engineering with limited deep theory, which explains why progress is driven more by practitioners than by pure mathematicians.
  • Concerns are raised about deploying generative models with real-world agency without better internal understanding, likening them to complex biological or chaotic systems.
  • Others argue we already know enough to see their statistical, non-verifiable nature as the main safety constraint, insisting on human oversight for high-stakes uses.