Generative Models: What do they know? Do they know things? Let's find out

Original Article ↗ Hacker News Discussion ↗

Generative image models appear to implicitly learn 3D scene properties like depth, surface normals, albedo, and lighting, even though they are only trained on 2D images and text. Commenters see this as evidence that such systems build internal, human-interpretable representations rather than merely “gluing together” pixels, prompting debate over whether this counts as genuine world modeling or understanding versus sophisticated pattern-matching. The work also fuels speculation about applications in vision, VR, and 3D reconstruction, as well as broader questions about how closely these models resemble human perception and learning.

Access / Setup

  • Some users saw SSL issues on the main site; others confirmed the certificate and provided the GitHub mirror.
  • Fingerprint details of the cert were shared to show it is valid from at least some locations.

Internal 3D / Scene Representations

  • Many are struck that generative image models appear to implicitly represent depth, surface normals, albedo, and possibly material properties like specularity.
  • This is compared to traditional physically based rendering pipelines and deferred shading/G-buffers.
  • The fact that autoregressive and diffusion models, trained only on 2D images + text, can yield high-quality depth/normal maps surprises people and is taken as evidence of an internal “world model” or 3D engine.

“Understanding” vs Stochastic Parrots

  • One side argues these models clearly learn abstract, human-interpretable structures, undermining the “stochastic parrot” / mere-regurgitation view.
  • Others counter that there is “zero evidence” of genuine understanding and that models are still just optimizing to reproduce training data statistics.
  • A middle view: “understanding” is best thought of as having an internal causal model that generalizes; by that standard, it is a spectrum, not a binary.

Qualia, Consciousness, and Human Comparison

  • Some equate criticism of AI understanding with old debates about souls.
  • Others insist there is a fundamental gap: humans have qualia and will; models do not, and we don’t even know how we’d detect it.
  • There’s pushback that qualia may be ill-defined and possibly reducible to internal states and self-models; if so, systems like these might eventually qualify.

Skepticism About World Models

  • Critics argue that consistent 3D reasoning would eliminate artifacts (extra limbs, bad perspectives), which still occur.
  • Some note traditional tools can estimate normals/depth from a single image without claiming full world simulation.
  • It’s unclear whether the underlying UNet was ever trained on paired G-buffer data; some suspect that could be hidden “secret sauce.”

How I-LoRA Works (High-Level)

  • Explanations emphasize that I-LoRA adds a small number of trainable parameters (LoRA) on top of a frozen base model and fine-tunes only those.
  • Because <1% of parameters are changed yet good normals/depth emerge, commenters infer the base model already contains near-ready representations; LoRA merely “reads them out.”

Applications and Extensions

  • People foresee benefits for VR/spatial computing (stereo views), relightable 3D reconstructions (e.g., gaussian splatting), and better depth maps—though some note typical depth maps break under real use.
  • There is curiosity whether similar techniques could be applied to text models (e.g., editing factual knowledge) and to other generative systems, including NN-generating NNs.

Sora, World Simulation, and Hallucinations

  • Some link this work to reactions to Sora, taking both as evidence that video models run an implicit physical simulation rather than merely “gluing clips together.”
  • Others highlight glaring Sora artifacts (e.g., extra limbs) as reasons to doubt robust physical modeling.
  • Human hallucinations and dreamlike distortions are cited as an analogy: flawed outputs don’t necessarily mean no world model.

LoRA Size and “Real Extraction” vs New Model

  • A concern is that a LoRA with millions of parameters might itself be doing most of the work, creating plausible normals/depth rather than truly exposing existing structure.
  • Supporters point to controls: when LoRA is trained on a randomly initialized UNet, intrinsic maps collapse, suggesting the pre-trained model’s features are essential.

Real-Image Use and Depth / Normal Extraction

  • The paper’s claim that diffusion models are image-to-image structures makes applying them to real photos straightforward.
  • Depth and normals from real photos are demonstrated; albedo/lighting from real photos remain speculative but are seen as especially exciting for relighting.

Relation to Broader Interpretability Work

  • Commenters connect this to other mechanistic interpretability results (e.g., internal structure in Othello-playing transformers, fact-editing in LLMs).
  • The emerging picture: large generative models encode rich intermediate representations that can be tapped with relatively light extra training.

Funding and Title / Pop-Culture Reaction

  • There’s amusement and some surprise at corporate funders (e.g., automotive and creative software companies) backing research with a whimsical, TV-inspired title.
  • The Bojack Horseman gameshow reference is widely appreciated, prompting discussion of the show’s mix of humor and bleakness and how that mirrors the paper’s playful framing of serious work.