I struggle to believe that AI R&D is ingenious

Skepticism is growing over claims that large language models can perform truly ingenious, autonomous research and development, rather than sophisticated pattern-matching over existing work. Commenters describe LLMs as powerful accelerators for grunt work, code scaffolding, literature review, and even some math proofs, yet consistently weak at judgment, system-level design, and genuinely novel ideas without heavy human-built “harnesses” and supervision. Others counter that internal frontier models, tool use, and agent swarms are already pushing past these limits in specialized domains, suggesting that scale and better training may keep extending capabilities even if current systems often miss “the forest for the trees.”

Perceived capabilities vs limitations of current LLMs

  • Many see LLMs as powerful assistants for grunt work: literature review, scaffolding experiments, diagnostics, visualization, code, and documentation.
  • They are often described as knowledgeable but “low judgment” – good at routine tasks, weak at big-picture reasoning and prioritization.
  • Several note that models impress non-experts but look shallow or error-prone to domain experts, echoing “Gell-Mann amnesia” dynamics.

Autonomous agents and harnesses

  • Users report poor results from leaving agents “to their own devices” on open-ended R&D problems.
  • Effective systems rely on elaborate harnesses: DAGs of many specialized sub-agents, carefully staged tasks, and tight human oversight.
  • Agent swarms can explore many ideas in parallel but show diminishing returns and require significant compute and engineering.

Creativity, novelty, and “stochastic parrot” debate

  • One side argues LLMs only remix existing text, producing bland, average outputs and failing at “pure creativity.”
  • Others counter that most human creativity is also recombination; models can still yield ideas that are novel to humans (e.g., jokes, names, research tricks).
  • Several note that “novel” usually arises when humans carefully constrain style, context, or cross-domain mappings.

Math breakthroughs and proof search

  • The recent LLM-assisted proofs (e.g., major results in number theory and complexity) are a central point of contention.
  • Supporters say these show genuine high-level problem solving, not naive brute force; they compare it to thousands of PhD-level mathematicians exploring ideas.
  • Skeptics cite heavy tool use (Lean, search loops), huge compute, low success rates per attempt, and inelegant or opaque proofs. Some fear progress without understanding.

Expertise, context, and judgment

  • Commenters highlight failures in tracking system state, basic logic, design aesthetics, and architecture-level “vision.”
  • LLMs excel when tasks are well-posed, objectively checkable, and richly represented in training data (e.g., standard coding, some ML, some math).
  • They struggle with fuzzy goals, nuanced tradeoffs, or tasks where “not missing the forest for the trees” lacks a clear training signal.

Scaling, internal models, and future trajectory

  • Some believe frontier labs’ internal models are far ahead and already enabling rapid autonomous R&D; others view such claims as unverified PR.
  • There is disagreement on whether we’re near a ceiling (diminishing returns on scale, persistent qualitative gaps) or still early in an exponential phase.
  • Several frame progress as a “bitter lesson”: brute-scale search and RL may matter more than human ingenuity in AI R&D, even if models themselves remain non-ingenious.