A walk through of the DeltaNet family of linear attention variants
A technical blog post claiming “You Could Have Come Up With Kimi Delta Attention” prompts mixed reactions, with many readers feeling the math and notation are far beyond what the title implies. Commenters debate the accessibility of modern ML research, the role of specialized notation like bra–ket versus more explicit code-like formulations, and the extent to which breakthroughs in transformer architectures and linear attention depend on conceptual insight versus access to large-scale compute. Several point out that while the underlying operations are “just” linear algebra, the information density, background knowledge, and fast-moving nature of the field make it unrealistic for most practitioners to independently reinvent such methods.
Overall reaction to article and title
- Many readers found the “you could have come up with…” framing irritating or unintentionally condescending, especially given the technical density.
- Some note this title style is a known expository genre meant to build intuition, but argue it only works if the piece actually makes the result feel reachable.
- Others treat it as harmless rhetoric and appreciate that the post walks through incremental steps.
Difficulty, accessibility, and notation
- A large subset admits they could not have come up with the method and struggled even with the notation.
- The bra–ket vs “normal math” toggle is widely praised; some find bra–ket clarifying, others say it’s unnecessary and intimidating.
- There’s broader frustration with dense, inconsistent math notation in ML, versus a desire from some for more code‑like or verbose notation.
- Several comments emphasize that math/ML texts are inherently high‑information density and must be read slowly and symbol by symbol.
Understanding transformers and attention
- One commenter gives a simple mental model: each token prediction is one big function call through stacked transformer blocks, repeated token by token.
- Clarifications around layers, hyperparameters, and keys/queries/values help some readers connect the math to implementation.
Linear attention and DeltaNet‑style variants
- Readers appreciate the step‑by‑step derivation and especially the key trick of accumulating outer products into a fixed‑size state.
- Some wanted more intuition on why such architectures work at all, noting they resemble compressing history into a fixed hidden state like RNNs/LSTMs.
- There’s recognition that all linear‑attention schemes approximate full quadratic attention and likely face trade‑offs in expressiveness.
Innovation, compute, and who can contribute
- Debate over whether “you could have invented this” is realistic:
- One side: many breakthroughs are constrained more by access to vast compute than by pure math insight.
- Another side: the hard part is seeing simple ideas before they’re obvious in hindsight.
- Several reflect on human cognitive limits, the need for very smart people even in an AI‑rich future, and how expertise shapes what feels “obvious.”
Getting started and practicalities
- For newcomers with linear algebra background, commenters suggest small models, consumer GPUs or even CPU, and beginner‑friendly video series.
- Hardware is seen as a non‑issue for learning basics, though crucial for frontier‑scale experiments.