GoKawiilMath notation ⟨k|q⟩ kᵀq
A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ \lvert q\rangle ∣q⟩ is a column vector, ⟨ k ∣ \langle k\rvert ⟨k∣ is a row vector, ⟨ k ∣ q ⟩ \langle k\rvert q\rangle ⟨k∣q⟩ is a number, and ∣ v ⟩ ⟨ k ∣ \lvert v\rangle\langle k\rvert ∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.
Modern linear attention variants are complex, and a upon first glance it is not so easy to see what they are designed to achieve. For reference here is the state update equation for Kimi Delta Attention (KDA):
S ~ t = S t − 1 Diag ( α t ) \widetilde S_t = S_{t-1}\operatorname{Diag}(\alpha_t) S t = S t − 1 Diag ( α t )
∣ v ^ t ⟩ = S ~ t ∣ k t ⟩ \lvert\widehat v_t\rangle = \widetilde S_t\lvert k_t\rangle ∣ v t ⟩ = S t ∣ k t ⟩
∣ e t ⟩ = β t ( ∣ v t ⟩ − ∣ v ^ t ⟩ ) \lvert e_t\rangle = \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right) ∣ e t ⟩ = β t ( ∣ v t ⟩ − ∣ v t ⟩ )
S t = S ~ t + ∣ e t ⟩ ⟨ k t ∣ S_t = \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert S t = S t + ∣ e t ⟩ ⟨ k t ∣
∣ o t ⟩ = S t ( d k − 1 / 2 ∣ q t ⟩ ) \lvert o_t\rangle = S_t\left(d_k^{-1/2}\lvert q_t\rangle\right) ∣ o t ⟩ = S t ( d k − 1/2 ∣ q t ⟩ )
The reason they are so difficult to understand is that this is the latest in a family of linear attention variants that have been developed over the last few years and the complexity of them has inevitably ballooned such that from the outside the latest variants appear inaccessible.
In this post we are going to walk through the DeltaNet family of linear attention variants, two of which are used by the latest Qwen and Kimi model families, and show how you might have arrived at the same equations by asserting simple things about your hidden state.
... continue reading