There are no new ideas in AI, only new datasets

Original Article ↗ Hacker News Discussion ↗

Role of Data vs Methods

  • Many argue recent AI gains mostly come from larger, cleaner, and more diverse datasets plus more compute, not fundamentally new algorithms.
  • Others counter that architectural ideas (transformers, attention, long context, multimodal models, RL from human feedback, video world models) are substantive innovations, even if rooted in old math.
  • Several note a cyclical pattern: new ideas → heavy scaling (data/compute) → diminishing returns → renewed search for new ideas.

Hardware, Infrastructure, and Timing

  • Commenters stress that 1990s-era ideas only became practical due to:
    • Massive GPU/parallel compute.
    • Cloud-scale infrastructure and fast networks.
    • The internet as a text-centric, labeled-data firehose.
  • Some predict current architectures will hit a “compute coffin corner” where costs grow faster than quality, triggering an “AI winter”; others expect a plateau of productivity instead.

Generalization, Reasoning, and “Real” Intelligence

  • RL/game examples: models can reach superhuman skill on one game yet fail to transfer to new levels or similar games, sometimes performing worse after prior training—seen as overfitting and weak abstraction.
  • Debate over whether LLMs just memorize training data or truly generalize:
    • One side: they are sophisticated pattern-matchers/compressors producing plausible outputs, not reasoning.
    • Other side: they clearly solve novel tasks and manipulate unseen codebases, which implies some form of reasoning, even if alien to human cognition.
  • Symbolic AI and meta-RL are cited as alternatives that sometimes show better true generalization.

Embodiment, Multimodality, and World Models

  • One camp: language+vision already capture the core of human-level online intelligence; touch, smell, etc. mainly matter for robotics.
  • Opposing camp: embodiment and rich sensorimotor experience are foundational for robust world models (space, causality, physical intuition); text/video alone are “shadows in a cave.”
  • Robotics, simulation, and video world-models are seen as next frontiers, but current video models still lack object permanence and robust physics understanding.

Static Training vs Continuous Learning

  • Current LLMs are static, not always-on learners; they don’t reliably update weights in real time or manage long-term memory/forgetting.
  • Some see reinforcement learning, evolutionary methods, self-play, and agentic systems as paths toward continuous adaptation, but note they are compute-expensive and brittle.

Tool Use, Data Limits, and Future Directions

  • A recurring wish: models that reliably delegate precise subproblems to traditional algorithms and tools without bespoke wiring for each task.
  • Concern that web-scale high-quality text is close to exhausted; suggestions include biological, robotics, and video data, plus more synthetic/simulated experience.
  • Several insist new architectures are being explored constantly, but most yield incremental gains; data quality/diversity often matter more in practice.