GoKawiilOpenArch open-sources readable PyTorch rewrites of major LLM architectures
A new open-source repository called OpenArch offers hand-written PyTorch implementations of prominent LLM architectures, including GPT-2 XL, Llama 2, Llama 3, OLMo 2, DeepSeek R1 and Gemma 3. Each model is coded from scratch based on original papers, technical reports and config files, with one clear, self-contained file per architecture rather than production-optimized code.
GoKawiil's interpretation of the reporting above, not reported fact.
Production ML codebases prioritize speed and hardware efficiency, often making it hard to see how core design choices—attention variants, normalization schemes, positional encodings, and MoE routing—actually differ across models. By stripping away those optimizations, OpenArch gives researchers and students a side-by-side reference for understanding how modern LLMs are architecturally built, which could accelerate both education and experimentation with new architecture combinations.
- OpenArch provides from-scratch PyTorch code for models like Llama 2/3, DeepSeek R1, Gemma 3 and OLMo 2.
- The project prioritizes readability and comparison over performance, unlike production libraries such as transformers.
- It documents key architectural variations including attention types, normalization methods, and MoE routing across major LLMs.
Source: github.com, 2026-09-14
Published there as: “OpenArch – PyTorch implementations of modern LLM architectures”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.