Why AI systems don't learn – On autonomous learning from cognitive science

Autonomous vs Offline Learning

• Many comments agree current mainstream models mainly do offline learning on static, human-curated data, not true autonomous learning via ongoing interaction.
• The paper’s critique of a “data wall” and “padded room” training (isolation from the real world) resonates with several commenters.
• Others argue that once LLMs help generate, filter, and label their own training data, we are already partway to self-training systems.

Meta-Control, System A/B/M, and Implementation Challenges

• The A/B/M framework (observation, action, meta-control) is seen as conceptually appealing but implementation details are viewed as the hard part.
• Concerns that agents could create self-reinforcing, hallucinated feedback loops when learning from their own actions.
• Questions arise about how to design reward signals for switching between passive observation and active exploration without collapsing into one mode.
• Some suggest we may need additional “systems” beyond neural networks (analogous to emotions/hormones) to manage this meta-control.

Ethics, Machiavellian Behavior, and Anthropomorphism

• One line of discussion worries that truly autonomous corporate agents could become ruthlessly Machiavellian, outcompeting human bad actors.
• Others counter that algorithms lack intrinsic morality; any apparent ethics or manipulation is just behavior shaped by objectives and data.
• ELIZA and the “ELIZA effect” are invoked to explain both over-anthropomorphizing current systems and investor/“AI hype” dynamics.
• In contrast, another thread cites the “AI effect” as humans moving the goalposts whenever machines master a previously “intelligent” task.

LLM Capabilities, Cognition, and In-Context Learning

• Strong disagreement over whether LLMs “actually learn”:
  • One side: they only fit data offline; tools, RAG, and filesystems are just pre-programmed mechanisms, not cognition.
  • Other side: LLMs plus external memory and tools form systems that, at the system level, exhibit learning-like behavior.
• Debate on whether cognition requires online weight updates vs. being realizable via context, memory stores, and agents.
• Some think the paper underplays in-context learning and real-world agent architectures; others think expectations for LLMs are delusional.

Online Learning, Safety, and Product Concerns

• Historical example of a Twitter-trained bot rapidly degenerating into toxic speech is used to argue that “not learning online” is a safety feature.
• Production teams prefer fixed, versioned models over continuously self-modifying systems, to maintain predictability and control.
• Tension noted between:
  • Desire for systems that “learn on the job” (e.g., proprietary codebases, domain expertise), and
  • Fears about data leakage, unpredictable behavior, and misalignment if models freely update from user inputs.

World Models, JEPA, and Compute Constraints

• Interest in “world models” that learn physics and dynamics via interaction, not just text ingestion.
• Skepticism that such models can be trained with current budgets; physical interaction data is seen as more unstructured and compute-hungry than internet text.
• Some expect large LLM-first labs, funded by LLM revenue, to eventually build the kind of world models envisioned in the paper.

Cybernetics and Broader Inspiration

• Several see current discussions as rediscovering mid-20th-century cybernetics: feedback, control, and system-level thinking.
• Others find cybernetics historically “wishy-washy,” unclear how much concrete, lasting technical substance it contributed vs. inspiring later fields.
• Biological and synthetic-biology-inspired hardware is mentioned as a possible future route to truly learning, brain-like systems, but remains speculative in the thread.

Diversity, Forked Models, and Evolutionary Ideas

• Some advocate for many diverse, personalized models that continue to learn, rather than a few homogeneous, frozen systems.
• Arguments: diversity reduces shared vulnerabilities (memetic or otherwise) and might drive creativity and capability via selection-like processes.
• Others worry that uncontrolled online learning risks “model collapse,” safety issues, and unpredictability.

Meta-Level: What Counts as “Real AI”?

• Persistent meta-debate:
  • One side sees current systems as close to matching or exceeding average humans on many “intelligent” tasks, with remaining gaps not clearly fundamental.
  • The other side insists the key unsolved issues (online learning, robust reasoning, new problem-solving) are precisely what “real intelligence” requires.
• Both hype (“AI is here”) and dismissal (“these are just parrots”) are criticized; several commenters call for more careful, system-level definitions of learning and cognition.