How far are we from intelligent visual deductive reasoning?

Original Article ↗ Hacker News Discussion ↗

Current visual LLM capabilities

  • GPT-4-Vision and similar models perform surprisingly well on practical tasks like tagging and captioning images.
  • They can objectively describe people in photos when steered away from subjective or judgmental language, though safety filters can be “fussy.”
  • On complex scenes (e.g., truck with porta-potties under a low bridge, convertible behind), models can detect danger and sometimes the slapstick outcome, but often need strong hints and may miss key elements (cargo, overpass, humor).

Limits of visual deductive reasoning

  • Models tend to answer even when uncertain, often producing confident but nonsensical rationales rather than admitting ignorance.
  • They often fail to grasp the full interaction between objects or the implicit narrative (“what’s about to happen?” or “what’s funny here?”).

Confidence, correctness, and learning theory

  • Discussion of how neural nets might estimate “confidence”: output token probabilities, agreement among subnetworks or repeated passes, or external fact-checking.
  • Several comments argue that, by design, probabilistic models (PAC/BPP) can’t reliably know when they’re wrong in the general case.
  • Disagreement over using “Dunning–Kruger” as a label; some say it’s a misapplied metaphor.

Architectures and multimodality

  • Current vision–language models are seen as “Frankenstein” systems with separate vision modules feeding embeddings into an LLM.
  • Some argue for a single decoder-only transformer trained jointly on text and image tokens, expecting richer behaviors (SVG from logos, better ASCII art, more structured code from images).
  • Video remains hard: systems sample a few frames, unlike self-driving stacks that use optimized, task-specific pipelines.

Statistical vs symbolic intelligence

  • Strong debate on whether purely statistical systems can ever reach human-level intelligence.
  • Critics say current ML is mostly “sophisticated compression” of (Q, A) pairs, not genuine understanding or causal modeling.
  • Others counter that brains may also be enormous statistical systems over particles; “everything could be statistics.”
  • There’s repeated calls for renewed work on knowledge representation, causal reasoning, hybrid symbolic–neural systems, and interpretable, argument-producing AI.

Benchmarks and task-specific solutions

  • Raven’s Progressive Matrices and the ARC benchmark: humans ≈80% vs best models ≈25%.
  • Simple, hand-crafted heuristics (e.g., row/column sums, XOR masks) can nearly solve some visual IQ tests in a few lines of code, suggesting these benchmarks can be “gamed” without human-like reasoning.
  • GPT-4 does better on certain abstract reasoning tasks when they are recast as 1D, text-based problems.
  • Specialized image-token prediction models can handle RPM-like tasks as inpainting, hinting that representation format matters greatly.

Self-driving and real-world implications

  • Some earlier skeptics of end-to-end learning for driving now see visual models’ emerging ability to predict “what happens next” as evidence it may eventually work.
  • Others note current autonomous driving failures and argue that safety-critical decisions (e.g., tradeoffs in collisions) require more than pattern recognition, raising ethical and design questions.

Creativity and applications

  • Early image models sometimes produced uncanny, non-human-like art; some lament that progress has pushed systems toward more “safe” face-swap–style images, though others say creative outputs are still possible.
  • Practical use cases include identifying ambiguous real-world objects (e.g., partially submerged vehicles) via large visual corpora.