LLMs aren't world models

Chess, Games, and “World Model” Claims

• OP’s examples (LLMs losing track of pieces, making illegal moves) are cited as evidence they lack even basic chess world models. Critics note that with dedicated chess training, small transformers can internalize full board state and play ~1800 Elo.
• Some argue that failure to reach near-100% move legality is damning, since legality is easy; others respond that even human amateurs rarely attempt illegal moves, so occasional illegality doesn’t imply “no model.”
• Papers and demos show SOTA LLMs now mostly play legal moves; skeptics reply that this often requires special training or tools and isn’t robust across models or prompts.

Math, Counting, and Internal Representations

• “Blueberry B-counting” and simple arithmetic failures are used as archetypal non-world-model behavior; others show current models answering these correctly, or suggest hard-coded fixes / RL patches.
• Interpretability work is invoked: internal neurons encode concepts like addition or board positions, suggesting some world-like structure.
• Critics of the essay say cherry-picked failures don’t outweigh evidence like gold-level performance on math Olympiads, which seems to require a transferable mathematical model.
• Defenders reply that success is narrow, heavily RL-tuned, and not compelled by next-token training; generalization often breaks on atypical problems.

Codebases, Autonomy, and Falsifiable Predictions

• A central claim: LLMs will “never” autonomously maintain large codebases; they can’t form stable internal models of novel systems without weight updates.
• Others point to tools like Claude Code / Cursor as early counterexamples, arguing hybrid LLM+tool agents already perform nontrivial multi-file work; but even fans concede they’re brittle and need expert supervision.
• Debate hinges on definitions: what counts as “large,” “deal with,” and “autonomous” (no human coders vs productivity aid).

World Models, Symbols, and Human Comparison

• Philosophical thread: language (and thus LLMs) manipulates symbols, not reality; “the map is not the territory,” so pure language models can’t be full world models.
• Counter-argument: all cognition (including human) is symbolic / representational; if neurons can encode a world model, so can sufficiently rich token-based systems.
• Several note humans also hallucinate, confabulate, and rely on external scaffolding (boards, notebooks); LLMs may need similar persistent memory and tool integration.

Hybrid Architectures and Future Directions

• Many commenters expect progress from hybrids: LLMs wrapped with deterministic tools, search, planners, or non-language world models (e.g., game/video models like Genie).
• Consensus-ish middle: LLMs are powerful but inconsistent generalists, with patchy and compressed world models; useful in practice, but not obviously the final route to AGI.