Extracting concepts from GPT-4

Original Article ↗ Hacker News Discussion ↗

OpenAI’s new work on “extracting concepts” from GPT‑4 using sparse autoencoders aims to make large language models more interpretable by mapping internal activations to human-understandable features, such as rhetorical questions or price changes. Commenters compare it closely to Anthropic’s recent interpretability research, debating how novel OpenAI’s contributions are, how far this actually goes toward understanding or controlling model behavior, and what it might mean for safety and future applications like feature-level steering or advanced semantic search. Many see it as promising but still very early progress toward unpacking what remains a largely opaque technology.

Overall reception

  • Many find the work exciting as a move toward “deep” semantic search and interpretable concepts inside GPT-4.
  • Others see it as incremental and still evidence that LLMs are largely black boxes, citing the article’s own statements about how little is understood.

Relation to prior interpretability work

  • Repeated comparisons to Anthropic’s sparse autoencoder / “Scaling Monosemanticity” work.
  • Some claim OpenAI is mostly copying; others argue the methods are concurrent and OpenAI introduces meaningful additions (e.g., new activation functions, dead-latent mitigation, new evaluations).
  • Several note Anthropic’s demos and visualizations feel more polished and “impressive,” while OpenAI emphasizes less-cherry-picked, more random features to avoid interpretability illusions.

What sparse autoencoders enable

  • Thread-wide explanation: models contain many “features” or “concepts” encoded in internal activations (from punctuation to historical facts to price changes).
  • SAEs are presented as a way to decompose these tangled activations into sparse, human-interpretable features.
  • This could allow:
    • Inspecting which concepts fire for given prompts.
    • Ablating or boosting concepts to study behavior or steer outputs.
    • Possibly manipulating specific knowledge or safety-relevant concepts without disrupting others.

Limits, skepticism, and open questions

  • Multiple commenters stress this is very early; we still lack a general understanding of how transformers work, why capabilities emerge, or how to fully debug them.
  • Debate over hallucinations: some think interpretability could eventually help; others argue LLMs are “always hallucinating,” and distinguishing fact vs. fiction internally may be ill-posed.
  • Some doubt whether we’ll ever get a low-level, brain-like understanding of such complex systems.

Safety, risk, and legal angles

  • View that better interpretability is crucial for safety (e.g., detecting deception, controlling harmful concepts).
  • Counterpoint that we already know the system “just outputs tokens,” and real risk lies in how people use those outputs.
  • Extended debate using analogies (knives, cars, Google Search, social media) about what should be regulated: underlying tech vs. applications.
  • Questions about training data: viewer uses The Pile as “uncopyrighted,” implying internal GPT-4 data is copyright-sensitive; some raise potential legal and “fair use” issues.

Potential applications and tooling

  • Ideas include semantic search based on concept activations, hybrid search with sparse features, caching “hot spots” to speed inference, browser extensions for knowledge workers, and better content filtering.
  • Open-sourced SAE code (on an older open model) and tokenization tools are noted as practical outputs.