GoKawiilDeveloper reimplements DeepSeek V3 architecture in 190 lines of pure JAX
A Show HN project recreates the DeepSeek-V3 model architecture—including multi-head latent attention, decoupled RoPE, DeepSeekMoE routing and multi-token prediction—in roughly 190 lines of pure JAX with no Flax, Equinox or PyTorch dependencies. The author pretrained a 36.2M-parameter version from scratch on a MacBook Air M5 using public-domain detective fiction, reaching a validation loss of 3.51 after 12,000 training steps over about 2.6 hours.
GoKawiil's interpretation of the reporting above, not reported fact.
The project demonstrates that core techniques from a frontier-scale model's technical report can be distilled into compact, readable code and trained on consumer hardware, which could make advanced architecture concepts more accessible for learning and experimentation. The comparison showing similar validation loss with and without the auxiliary-loss-free balancing bias suggests the technique evens out expert load without necessarily improving loss at this small scale, though this may not generalize to production-sized models.
- A simplified, dependency-free JAX implementation of DeepSeek V3's architecture was built in about 190 lines of code.
- The model (36.2M params, 13.6M active per token) was trained from scratch on a MacBook Air using public-domain detective fiction texts.
- Auxiliary-loss-free expert balancing evened out expert load significantly (5.7–6.9% vs 0.09–18.6% per expert) without changing validation loss.
Apple MacBook Air M2 — This project literally pretrains a DeepSeek-V3-style model from scratch on a MacBook Air, showcasing how capable Apple's silicon is for hands-on ML experimentation. A great lightweight machine if you want to tinker with JAX, run small models locally, or follow along with projects like this one.
See Apple MacBook Air M2 on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: github.com, 2026-11-14
Published there as: “Show HN: DeepSeek V3 in ~200 lines of pure Jax”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.