How do neural networks learn?

Reading/access and paywalls

• Many find the original site nearly unusable on mobile due to heavy, intrusive ads.
• Various ad-blocking and privacy tools (DNS-level blockers, browsers, reader modes) are discussed as workarounds.
• Some note the irony of “democratizing AI” while the referenced paper is paywalled; others link free arXiv versions.

Scope and substance of the paper

• Several suspect it might be another over-simplified “physics explains NNs” paper that only applies to idealized feed-forward nets.
• Others argue the work is narrower and more concrete: characterizing how deep networks develop “features” and how this relates to gradients and training dynamics.
• One summary highlights claimed explanations for pruning, grokking, and an associated kernel-based method that performs well on tabular benchmarks.
• There is mild surprise that the core quantity (outer product of weights) could be novel.

What it means to “understand how NNs work”

• Participants distinguish between:
  • Knowing the training algorithm (gradient descent) vs. understanding internal representations and behavior.
  • “How they work” (mechanism) vs. “what they learn” (features, circuits, abstractions).
• Proposed criteria for real understanding include:
  • Predicting which architectures/hyperparameters will work on given data.
  • Explaining initialization, weight changes from specific samples, and how to fix particular errors.
  • Providing prescriptive tools, not just descriptive stories.

Debates on understanding, explanation, and philosophy

• Long subthread on what “understanding” even means, invoking philosophical debates and whether poorly defined questions can still be fruitful.
• Some see “it’s just gradient descent / curve fitting” as trivially true but practically uninformative, akin to saying software is “just logic gates.”
• Others defend the reductionist view: NNs are error-minimizing function approximators; the mystery is overstated.

Emergent behavior and generalization

• Discussion of “emergent properties”: unexpected capabilities (e.g., translation) that appear when scaling models and data.
• Disagreement over how much these depend on initialization/order of training vs. largely on scale and data.
• Generalization, brittleness to data shifts, and the inability to formally verify behavior are highlighted as core open issues.

Neural Feature Matrix (NFM) as a tool

• The NFM (WᵀW per layer) is viewed as:
  • Proportional to average gradient outer products over training data.
  • A potential metric on input space reflecting which directions the network “cares about.”
• Some see the proportionality as almost obvious; others emphasize the value is in what the NFM enables:
  • Tracking feature importance over training.
  • Explaining pruning and grokking patterns.
  • Driving a kernel method that iteratively updates its kernel via gradient statistics.
• There is debate over whether a static, global metric like WᵀW can faithfully reflect curved data manifolds, or whether it’s still a useful, interpretable approximation.

Interpretability, safety, and engineering vs theory

• Several note we “know” the math and code, yet lack insight into specific learned models (especially large transformers).
• Some frame modern NN work as largely empirical engineering with limited deep theory, which explains why progress is driven more by practitioners than by pure mathematicians.
• Concerns are raised about deploying generative models with real-world agency without better internal understanding, likening them to complex biological or chaotic systems.
• Others argue we already know enough to see their statistical, non-verifiable nature as the main safety constraint, insisting on human oversight for high-stakes uses.