Neural networks are clean
A longform interpretability essay argues that modern neural networks are far more “clean” and legible than once thought, with most neurons encoding single, coherent concepts rather than arbitrarily mixing unrelated ones. Commenters probe whether this overturns the popular “superposition” view, debate the reliability of techniques like sparse autoencoders, and question how far current methods really get us toward mechanistic understanding and AI alignment. While many see the work as an exciting step that could make it easier to inspect and steer large models, others caution that emergent behavior, unverifiable explanations in unfamiliar domains, and unresolved alignment challenges limit how transformative it may be.
Meaning of “clean”
- Commenters struggle with the term; proposed meanings include:
- Neurons mostly encode a single, human-legible concept rather than many unrelated ones.
- Networks do not arbitrarily “smoosh” concepts together, so internal structure isn’t hopelessly messy.
- “Clean” is framed as the opposite of “alchemy” or “black box” behavior.
- Some argue the essay never crisply defines “clean,” and that it often blurs “describing” behavior vs truly “explaining” mechanisms.
Interpretability claims in the post
- The article argues that many classic examples of polysemanticity/superposition were misread.
- What looked like mixed “dog + car” neurons often turn out to be more abstract features (e.g., V-shapes) reused across domains.
- With larger, modern models, most neurons in practice appear to have one main job plus related variants.
- Directly inspecting neurons and circuits (rather than relying on extra models like sparse autoencoders) is promoted as a better path.
- Some see this as strong evidence that interpretability scales better than previously thought and that many neurons are surprisingly legible.
Challenges, skepticism, and limits
- Several posters argue:
- Sparse autoencoders and many mech-interp tools act more like elaborate probing; they don’t guarantee faithful mechanisms.
- Post‑hoc neuron descriptions in familiar domains (dogs, cars) might not generalize to unfamiliar domains (e.g., brain imaging).
- Understanding individual neurons doesn’t automatically yield understanding of emergent, circuit‑level behavior.
- A nontrivial fraction of neurons/circuits remains unexplained; those could be where the hardest or most “cognitive” behavior lives.
- Some invoke undecidability/Rice’s theorem or “emergence” to argue that full mechanistic understanding may be unattainable or practically irrelevant.
- Others counter that such theoretical objections are overapplied and don’t preclude substantial practical insight.
Implications for alignment and control
- Optimistic view: if neurons and circuits are mostly clean, we can:
- Map concepts like honesty, deception, or dangerous knowledge.
- Steer or ablate specific behaviors by targeting identifiable circuits.
- Skeptical view:
- Alignment is not just about finding concept neurons; human values are underspecified and contested.
- Even perfect interpretability might show that undesirable behaviors are straightforward combinations of necessary knowledge, making them hard to remove without crippling capabilities.
Model structure, compression, and representation
- Discussion links “clean” features to:
- Larger models needing less extreme compression, so they generalize into broader but still interpretable concepts instead of amorphous mixes.
- Transformers being more analyzable than neuroevolution‑designed monoliths, due to known topology and redundancy.
- Viewing models as lossy compressors: features become semantic buckets that minimize loss, not arbitrary smears.
Meta: presentation and accessibility
- Many praise the post as fascinating and visually rich; others criticize:
- Extreme length, redundancy, and jargon.
- Heavy, laggy web design (auto‑playing 3D) that hinders reading.
- Lack of a concise, paper‑style structure and clear TL;DR, prompting community-made abridged and audio versions.