Why AI systems don't learn – On autonomous learning from cognitive science

Original Article ↗ Hacker News Discussion ↗

Autonomous vs Offline Learning

  • Many comments agree current mainstream models mainly do offline learning on static, human-curated data, not true autonomous learning via ongoing interaction.
  • The paper’s critique of a “data wall” and “padded room” training (isolation from the real world) resonates with several commenters.
  • Others argue that once LLMs help generate, filter, and label their own training data, we are already partway to self-training systems.

Meta-Control, System A/B/M, and Implementation Challenges

  • The A/B/M framework (observation, action, meta-control) is seen as conceptually appealing but implementation details are viewed as the hard part.
  • Concerns that agents could create self-reinforcing, hallucinated feedback loops when learning from their own actions.
  • Questions arise about how to design reward signals for switching between passive observation and active exploration without collapsing into one mode.
  • Some suggest we may need additional “systems” beyond neural networks (analogous to emotions/hormones) to manage this meta-control.

Ethics, Machiavellian Behavior, and Anthropomorphism

  • One line of discussion worries that truly autonomous corporate agents could become ruthlessly Machiavellian, outcompeting human bad actors.
  • Others counter that algorithms lack intrinsic morality; any apparent ethics or manipulation is just behavior shaped by objectives and data.
  • ELIZA and the “ELIZA effect” are invoked to explain both over-anthropomorphizing current systems and investor/“AI hype” dynamics.
  • In contrast, another thread cites the “AI effect” as humans moving the goalposts whenever machines master a previously “intelligent” task.

LLM Capabilities, Cognition, and In-Context Learning

  • Strong disagreement over whether LLMs “actually learn”:
    • One side: they only fit data offline; tools, RAG, and filesystems are just pre-programmed mechanisms, not cognition.
    • Other side: LLMs plus external memory and tools form systems that, at the system level, exhibit learning-like behavior.
  • Debate on whether cognition requires online weight updates vs. being realizable via context, memory stores, and agents.
  • Some think the paper underplays in-context learning and real-world agent architectures; others think expectations for LLMs are delusional.

Online Learning, Safety, and Product Concerns

  • Historical example of a Twitter-trained bot rapidly degenerating into toxic speech is used to argue that “not learning online” is a safety feature.
  • Production teams prefer fixed, versioned models over continuously self-modifying systems, to maintain predictability and control.
  • Tension noted between:
    • Desire for systems that “learn on the job” (e.g., proprietary codebases, domain expertise), and
    • Fears about data leakage, unpredictable behavior, and misalignment if models freely update from user inputs.

World Models, JEPA, and Compute Constraints

  • Interest in “world models” that learn physics and dynamics via interaction, not just text ingestion.
  • Skepticism that such models can be trained with current budgets; physical interaction data is seen as more unstructured and compute-hungry than internet text.
  • Some expect large LLM-first labs, funded by LLM revenue, to eventually build the kind of world models envisioned in the paper.

Cybernetics and Broader Inspiration

  • Several see current discussions as rediscovering mid-20th-century cybernetics: feedback, control, and system-level thinking.
  • Others find cybernetics historically “wishy-washy,” unclear how much concrete, lasting technical substance it contributed vs. inspiring later fields.
  • Biological and synthetic-biology-inspired hardware is mentioned as a possible future route to truly learning, brain-like systems, but remains speculative in the thread.

Diversity, Forked Models, and Evolutionary Ideas

  • Some advocate for many diverse, personalized models that continue to learn, rather than a few homogeneous, frozen systems.
  • Arguments: diversity reduces shared vulnerabilities (memetic or otherwise) and might drive creativity and capability via selection-like processes.
  • Others worry that uncontrolled online learning risks “model collapse,” safety issues, and unpredictability.

Meta-Level: What Counts as “Real AI”?

  • Persistent meta-debate:
    • One side sees current systems as close to matching or exceeding average humans on many “intelligent” tasks, with remaining gaps not clearly fundamental.
    • The other side insists the key unsolved issues (online learning, robust reasoning, new problem-solving) are precisely what “real intelligence” requires.
  • Both hype (“AI is here”) and dismissal (“these are just parrots”) are criticized; several commenters call for more careful, system-level definitions of learning and cognition.