Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

Original Article ↗ Hacker News Discussion ↗

Kimi Linear presents an open-source, more efficient attention architecture that underpins Moonshot’s newer Kimi K3 model, which adds large-scale improvements like Stable LatentMoE, native vision, and extensive reinforcement learning. Commenters highlight rapid cross-pollination between architectures such as DeltaNet, Mamba, and RNN-like designs, and note that much of the “intelligence” in frontier models appears to emerge from sheer scale and well-chosen training objectives rather than a single breakthrough idea. Alongside the technical praise, there is a contentious debate over model distillation, intellectual property, and differing regulatory and censorship norms between Chinese and Western AI labs.

Relationship to Kimi K3 and Kimi Linear

  • Kimi K3 is described as heavily based on Kimi Linear, scaling it up and adding vision, RL enhancements, and Stable LatentMoE.
  • K3 uses “Kimi Delta Attention” layers, but the specific KDA variant differs from the one in Kimi Linear; these differences are said to be detailed in the K3 technical report.
  • Some note that Kimi Linear’s paper is from the previous year and that many contributions in such models are made well before product releases.

Architecture Evolution (DeltaNet, Gated DeltaNet, RNN lineage)

  • Commenters see Kimi Linear and related works (DeltaNet, Gated DeltaNet, Mamba-like models) as descendants of RNNs/LSTMs, reformulating attention as recurrent updates.
  • Gated DeltaNet 2 is reported to outperform Kimi Linear in internal tests, with Kimi Linear framed as an earlier step in this progression.
  • There is appreciation for how quickly companies iterate on and integrate each other’s architectural ideas.

Stable LatentMoE and Training Strategy

  • The main innovation credited to K3 is Stable LatentMoE: compressing inter-layer data with a more balanced expert routing strategy.
  • Thread notes that most recent progress comes from late-phase RL training: multiple expert models, sandboxed tools, human preference data, and agent traces.

Distillation, IP, and Ethics

  • Large portion of discussion debates “distillation attacks”: training one model on another model’s outputs.
  • One side: this is framed as an “attack” and unfair free-riding on expensive training and paid data licenses; especially sensitive in a China–West competitiveness context.
  • Other side: argues this is analogous to how LLMs already train on human-produced (often unpaid) data; sees hypocrisy in demanding IP protection for models but not for underlying human data.
  • Several question the “attack” language, suggesting neutral terms like “distilled variants” and viewing ToS violations as civil, not criminal, issues.

Censorship and Model Guardrails

  • Some users claim Chinese models are less censored, especially when self-hosted; others point to examples of political hedging or API-level restrictions.
  • General view: most “censorship” appears to be enforced at API/service layers rather than in base model weights.

Scaling, Emergent Abilities, and Theory

  • Multiple comments link frontier model “intelligence” to scaling laws and the “Bitter Lesson”: general-purpose learning plus more compute beats handcrafted algorithms.
  • Emergent abilities (e.g., complex reasoning, chain-of-thought) are discussed as:
    • Arising as models gain enough capacity to represent richer internal circuits.
    • Reflecting decreasing “excess entropy” with scale and data (“slingshot generalization”).
    • Appearing sudden on specific benchmarks but built on gradual improvements in generic capabilities.
  • Some compare this to function approximation, grokking, gradient descent, and search/hill-climbing methods.

Security and HTTPS Side Discussion

  • Short tangent on accessing an influential essay over HTTP vs HTTPS.
  • One side downplays risk for static text; another emphasizes MITM/script-injection threats even for seemingly simple pages.

Open-Sourcing and Ecosystem Impact

  • Kimi Linear’s open-sourcing of kernels, vLLM integration, and checkpoints is praised as “awesome” and unusually detailed compared to some Western labs’ communications.
  • Speculation that non-standard architectures (linear attention, RNN-like models) gaining traction could challenge hardware and compiler stacks optimized for standard transformers; implications for specialized hardware vendors are raised but not resolved.

Open Questions and Unclear Points

  • A direct question on how Kimi Linear performs on long-context retrieval (needle-in-haystack, ruler tests) versus full attention remains unanswered.
  • The exact quantitative contribution of distillation vs. novel data/architecture to Chinese models’ performance is acknowledged as unclear.