LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

Research focus: what the paper is actually claiming

• Commenters stress the work is about locating where relational facts are stored, not just verifying that the model “knows” them.
• The method fits linear functions that map internal states to specific attributes (e.g., “capital of X”), finding these work for many but not all relations.
• Some see this as modest but solid mechanistic insight; others think the press release oversells it and that results are partial and layer-limited.

Linear functions, equations, and embedding geometry

• Several comments clarify “linear function” in higher dimensions (e.g., Mv = c), not just y = mx + b.
• Discussion covers how embeddings live in high-dimensional spaces, with linear relationships supporting classic examples like “king – man + woman ≈ queen.”
• There’s debate about how “curved” internal decision surfaces really are, and why aggressive quantization still works.

Knowledge, memory, and hallucinations

• One interpretation: hallucinations can arise even when the model internally “has” a fact, if retrieval pathways fail.
• Others argue hallucinations are inseparable from the lossy compression nature of LLMs.
• Attention’s Q–K–V mechanism is discussed as a soft key–value store; some agree it’s KV-like, others see “lookup” as an overreach.

Compression, parameters, and the size of the internet

• Multiple comments frame LLMs as extreme text compressors, but clearly lossy.
• There’s back-and-forth on how much of “the internet” is actually seen in training and how much unique information exists after optimal compression.
• Clarifications distinguish parameter count from vocabulary size, and explain embeddings vs discrete tokens.

Lookup tables vs understanding and semantics

• One camp insists LLMs are sophisticated pattern matchers approximating text distributions, not semantic reasoners, and cannot ground meaning in reality.
• Another camp notes statistical models can recover useful hidden structure and that, in practice, rich learned representations can approximate semantic theories.
• A long subthread debates whether semantics can emerge from purely statistical learning or requires embodied, experimental interaction.

Architectures, local optima, and future directions

• Some argue transformers “won” after extensive experimentation; others say they may be a local maximum favored by current hardware and tooling.
• Alternatives (e.g., RNN-like or spiking models) and neuroevolution are mentioned as underexplored.
• Speculation appears on: directly writing facts into models, penalizing memorization to encourage reasoning, external memory (Paged/landmark attention), and compressing models via relation sets, but these remain open and largely speculative in the thread.