GoKawiilTL;DR
When DeepSeek-V4.1 Flash was released, I thought it might just be a post-training iteration version... but after using it for a while, I found it reached nearly 420 Tokens/s in speed, and then Cui said all DeepSeek-V4 Pro models would be taken offline... suddenly I felt this was no small matter... until the Technical Report was fully released, only then did I realize it should be called DeepSeek-V5 Flash...
As the paper title states, the purpose of DeepSeek-V4.1 Flash is to push KVCache compression to the extreme. The main reason is that Long-horizon Agent Workflows cause the Context to grow longer and longer, while various tool calls bring heavy prefill computation pressure. The storage pressure of KVCache in HBM and external SSD is very high, all of which are reasons that make Scaling impossible. Therefore, a series of optimizations were made on the model architecture, especially in the compression of KVCache and the computation optimization of Prefill.
Prefill computation optimization : Drawing on YOCO, the entire model has 40 layers, and only 20 layers are needed during Prefill. Therefore, the Prefill activated parameters are only 8B, and the Decode activated parameters are 16B
KVCache compression: Engineering-wise, KVCache compression is divided into several dimensions: head count compression similar to GQA, then block-based compression like CSA, and the cross-layer compression of CSA2 in this paper. At the same time, the indexer computation of Sparse Attention is also optimized. Finally, there are some numerical precision optimizations, for example DS41F adopts FP4 KVCache.
Finally, under the premise of maintaining high-quality task completion by the model, KVCache is further compressed by 4x:
In addition, the original writing of the paper is somewhat complex, especially the description of CED. In fact, if we redraw a diagram centered on KVCache and combined with the perspective of computer architecture, it seems to become clear all at once. It can be seen as a kind of Recursive Transformer architecture, a way of modifying Q and reusing KV during the recursive process.
Regarding the Recursive Transformer architecture, you can refer to 《On the Future Transformer: Loops Are Not What You Need》. Next, we will conduct a detailed interpretation and analysis according to the chapter structure of the technical report. This article is the first in this series, analyzing the model architecture in detail, and the more critical content is in Chapter 3.
1. Overview
1.1 Why KVCache compression is needed
... continue reading