There Will Be a Scientific Theory of Deep Learning

Original Article ↗ Hacker News Discussion ↗

Researchers and engineers are debating whether deep learning can ever be grounded in a compact, predictive scientific theory, rather than remaining an empirical craft driven by scale, data, and heuristics. Many point to the historical role of GPUs, large datasets, and architectural innovations like CNNs and Transformers in making neural networks practical, while noting that their success still outpaces our mathematical understanding of why gradient descent and overparameterized models generalize so well. Others are skeptical that a theory comparable to those in physics is possible, arguing that messy training data, enormous model sizes, and computability limits may prevent a clean explanation of how modern systems learn and fail.

Overall reactions to the paper

  • Many find the survey engaging, comprehensive, and particularly value the open-problems section.
  • Others see the title as overconfident “flag planting,” but still consider the work useful as a synthesis and standardization of ideas.
  • Several note a gap between active theory research and public perception that “it’s all just a black box.”

History: why deep learning took off when it did

  • Key inflection points cited: AlexNet (2012) for vision, then attention and transformers for language.
  • Crucial enabling factors: GPUs, much larger curated datasets (e.g., ImageNet), and better software frameworks that made complex models practical.
  • Some argue transformers “could have existed earlier,” but most replies stress results at small scale would have been underwhelming or infeasible to train.

Architecture vs scale and data

  • Strong debate:
    • One camp emphasizes “bagillions of parameters” and data as the main driver (the “bitter lesson”).
    • Another stresses architectural inductive biases and optimizer interactions; not all scalable architectures work, and many design choices are the difference between success and failure.
  • Neural nets are compared to “learned kernels” with powerful compositionality; nonparametric and kernel methods hit limits at modern data scales.

Relation to brains, evolution, and biology

  • Some argue deep learning is quite unlike brains; others see evolution as an end‑to‑end optimization process that pretrains brain structure.
  • There is discussion of local vs global learning rules (predictive coding vs backprop), and whether the brain approximates gradient descent.

Can there be a real “theory of deep learning”?

  • Optimists draw analogies to statistical mechanics, information geometry, and information‑theoretic views (implicit regularization, compression, scaling laws).
  • Skeptics doubt we’ll get physics‑like theories because behavior depends massively on messy data and huge models; they question whether concentration‑of‑measure style simplifications apply.
  • Some raise computability concerns (Rice’s theorem, Turing completeness), but others argue typical feed‑forward nets are not Turing complete, so classical impossibility results may not apply.

Interpretability, hallucination, and failure prediction

  • Several emphasize that theory matters most for predicting failure modes, confidence, and hallucination.
  • OOD detection is seen as conceptually shaky; alternative approaches based on model misspecification are being explored but are currently expensive and niche.

Models vs other ML approaches

  • Neural nets dominate unstructured data (images, audio, text), where their inductive biases match data structure.
  • Tree-based methods (often with boosting) remain superior on tabular data due to more suitable inductive biases there.