The deep learning boom caught almost everyone by surprise

Original Article ↗ Hacker News Discussion ↗

Deep learning’s sudden dominance is traced to a convergence of old ideas and new enablers: Hinton-era neural nets and backpropagation, Nvidia’s programmable GPUs, and Fei-Fei Li’s massive ImageNet dataset, along with key tweaks like ReLU activations and better weight initialization. Commenters argue over how much of the boom was genuinely surprising versus historically inevitable, debating whether current systems are on a path to AGI or just powerful but narrow tools. They also contrast deep learning with older machine-learning and symbolic methods, raising concerns about explainability, data hunger, hype cycles, and academia’s tendency to reward novel theory over scalable, data-driven practice.

GPU history and “invention” debate

  • Strong disagreement over the claim that one company “invented the GPU.”
  • Some argue earlier hardware (3dfx, SGI, Evans & Sutherland) and pre‑CUDA accelerators already counted.
  • Others say early chips weren’t truly general-purpose or useful for deep learning, and that the term “GPU” itself was a later marketing definition.
  • Consensus: the history is gradual and messy; calling any single launch “the invention” is seen as oversimplified.

Three pillars of the deep learning boom

  • Many commenters endorse the triplet:
    • Scalable neural networks and backpropagation.
    • Massively parallel GPU compute, programmable via CUDA.
    • Large labeled datasets like ImageNet, enabled by crowd labeling.
  • Some add: better activations (ReLU and successors), initialization, normalization, residual connections, and optimizers were also critical.

Data, scaling, and changing ML culture

  • Several note that earlier eras had tiny datasets and far less compute; “more data” wasn’t always practically feasible.
  • Pre‑ImageNet culture emphasized clever models and priors over sheer scale, especially in academia.
  • Others push back on a quote that “people did not believe in data,” arguing practitioners always valued more data but were resource‑constrained.

Deep learning vs other ML methods

  • Some wonder what would have happened if similar resources had gone into SVMs, random forests, etc.
  • Replies: classical methods struggle to scale, are less flexible to compose, and were heavily explored from the ’90s to mid‑2010s.
  • A recurring theme: LLMs are often used for trivial tasks where simpler models would be cheaper and adequate.

AGI, intelligence, and common sense

  • Split between those who see current progress as making AGI within years feel “inevitable” and those who think present systems fall far short.
  • Skeptics stress missing “common sense,” sample efficiency, and robust causal reasoning; draw contrasts with animals’ rapid generalization.
  • Debates over whether evolution counts as “training data,” how much biology’s architecture matters, and whether scaling deep learning alone can reach general intelligence.

Surprise vs inevitability

  • Some researchers say the boom felt more like “finally crossing a known threshold” than a true surprise.
  • Others emphasize how counterintuitive many specific breakthroughs were and how few predicted the scale or speed of the current wave.