Rich Sutton on AI creativity and discovery
Claims that current generative AI can only remix existing data, not produce truly novel and useful discoveries, are provoking scrutiny of what “creativity” in machines actually requires. Commenters contrast pure text- or image-generators with systems that close the loop by testing outputs against reality (e.g., code that compiles, game moves that win) and selectively retaining successes, arguing that this evaluation-and-iteration harness is where genuine innovation emerges. The exchange highlights unresolved questions about whether large language models plus reinforcement-style feedback can move beyond their training distribution into original science, engineering, and art, or whether fundamental architectural changes are still needed.
Interpretations of the Core Argument
- Many readers parse the argument as: creativity = variation + evaluation + selective retention, and claim that bare generative models mostly do variation.
- Some think the critique only applies to pretraining-era LLMs, not to modern systems with reinforcement learning and tools.
- Others say the talk is less “anti‑LLM” and more a call to embed evaluation and retention directly into AI workflows.
Novelty vs Quality (“Good and Novel”)
- The claim that LLM outputs are either “novel or good, but never both” is widely challenged.
- Commenters argue random exploration can still land on high‑quality ideas; the premise that randomness and goodness are mutually exclusive is disputed.
- Several note that recombining learned “pieces” (styles, abstractions, procedures) can yield genuinely new, useful compositions.
Role of Evaluation, Feedback, and Harnesses
- Strong agreement that closed‑loop systems (generation + test + refinement) underpin real discovery.
- Coding agents, math systems with proof checkers, and systems using compilers or theorem provers are cited as examples where evaluation is “neurosymbolically closed” and AI makes novel, validated advances.
- Many emphasize that the “agentic harness” around LLMs—tools, tests, self‑play, RL with verifiable rewards—is crucial, and the talk underplays this ecosystem.
Limits, Training Dynamics, and RLVR
- Debate over whether reinforcement learning with verifiable rewards truly expands beyond the base model’s distribution or just “mode‑seeks.”
- Some highlight continual-learning ideas (e.g., periodically reinitializing underused neurons) as a way to maintain variation and plasticity.
- Others argue current planners and search procedures remain “dumb” and are the true bottleneck, not the models.
Human vs Machine Creativity and Evaluation
- Several reject any special metaphysical barrier preventing AI from evaluating or discovering; humans also rely on tools and real‑world experiments.
- Others stress that humans have built‑in goals, embodiment, and long evolutionary history shaping evaluation, which AI lacks.
- Art and emotion: some say AI can create art if art is defined by evoking experiences in a viewer; others insist human intent and lived experience are essential.
Broader Reflections and Skepticism
- Some appreciate the framing of creativity as variation–evaluation–retention but find the conclusions unsurprising or underspecified.
- Concerns arise about over‑reliance on authority and dismissal of new empirical evidence from modern LLMs.
- Several see today’s models as already highly useful “discovery assistants,” even if they never reach paradigm‑shifting scientific genius.