Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks

Original Article ↗ Hacker News Discussion ↗

A new paper comparing GPT‑4 and its multimodal variant to humans on abstract visual reasoning tasks finds that neither model matches human‑level abstraction, reigniting debate over whether large language models can truly “reason” or are just sophisticated pattern matchers. Commenters probe the study’s methodology, especially its use of Mechanical Turk workers as a human baseline, and note that GPT‑4 still struggles with spatial and grid-based problems despite strong performance on many text tasks. The exchange broadens into questions about embodiment, multimodal training, and whether current transformer-based systems are a sufficient path toward general intelligence or just one powerful component in larger AI architectures.

Can LLMs Reason or Just Pattern-Match?

  • One camp argues LLMs only do next-token prediction over text, lack internal models of external systems, and thus cannot truly reason—just echo patterns seen in data.
  • Others counter that this is not proven; reasoning may itself be a kind of pattern exploitation, and complex behaviors can emerge without being explicitly “designed in.”
  • Neural networks as universal approximators are cited to argue that, in principle, reasoning could emerge in transformers if representable in their function class.
  • A logic-sequence example shows GPT-3.5 giving a self-contradictory answer, while GPT-4 gets it right, used both as evidence of limitations and of rapid improvement.

Representation, Abstraction, and Embodiment

  • A recurring theme: current models operate in token/embedding space, not in grounded physical or spatial realities.
  • Several commenters think robust abstraction requires being able to “simulate” the problem domain, possibly in different internal spaces per task.
  • Others point to multimodal work (vision + language) and findings that internal LLM representations can align with visual encoders, suggesting partial grounding is already happening.
  • Spatial/grid tasks (like ConceptARC) are seen as particularly hard for text-trained systems, analogous to newly sighted humans struggling with visual shapes.

Evaluation Design and Human Baselines

  • Some question the paper’s use of Mechanical Turk “master workers” as a human benchmark, noting MTurk quality issues, potential LLM use by workers, and lack of clear variance data.
  • Others reply that filtering low-effort or bot-like workers is standard and necessary, and that the core result—humans ~90% vs GPT-4 ~33%—is still informative.
  • There is concern the tasks conflate abstraction with narrow visuo-spatial reasoning and that matrix-encoded grids are not directly comparable to human visual perception.

Prompting, Tools, and Composite Systems

  • Multiple commenters ask whether chain-of-thought prompting, tool use, or better prompt formats would significantly raise scores.
  • Some argue the more relevant question is not “Can a bare LLM reason?” but whether an LLM embedded in an agentic loop with memory, tools, and sensors can.

Usefulness, Hype, and Future Directions

  • Several distinguish between AGI-level reasoning and current utility: LLMs can already outperform many crowdworkers on some tasks, even if they fail abstract benchmarks.
  • Skeptics see LLM hype as akin to crypto exuberance; others stress they are already economically valuable and improving quickly, especially when combined with code, search, or robotics.