Mla - GoKawiil

Developer reimplements DeepSeek V3 architecture in 190 lines of pure JAX

A Show HN project recreates the DeepSeek-V3 model architecture—including multi-head latent attention, decoupled RoPE, DeepSeekMoE routing and multi-token prediction—in roughly 190 lines of pure JAX with no Flax, Equinox or PyTorch dependencies. The author pretrained a 36.2M-parameter version from scratch on a MacBook Air M5 using public-domain detective fiction, reaching a validation loss of 3.51 after 12,000 training steps over about 2.6 hours.

github.com · 2026-11-14

DeepSeek's KV cache compression techniques adopted industry-wide, cutting AI serving costs

DeepSeek has published a series of KV cache optimizations—including MLA architecture, Compressed Sparse Attention, and now CSA2 with cross-layer reuse and FP4 caching—that reportedly shrink cache memory footprint per token by roughly 437x compared to its earlier V1 model. These techniques reduce the VRAM needed to serve long-context sessions such as coding assistants, and the report says other labs have begun using similar approaches without crediting DeepSeek.

insufferable.dev · 2026-09-30