GoKawiilDeveloper publishes open-source C++/CUDA LLM inference engine with 85-page handwritten notes
An independent developer has released a project that builds a transformer inference engine from scratch in C++17 and CUDA, documented through over six hours of video and an 80-plus page PDF of handwritten notes. The system currently loads Llama 3.2 1B Instruct weights directly from safetensors files, runs a custom Unicode-aware BPE tokenizer, and executes full dense causal prefill across all 16 layers including RoPE, grouped-query attention, SwiGLU MLP, and RMSNorm kernels to produce next-token predictions via GPU argmax.
GoKawiil's interpretation of the reporting above, not reported fact.
The project offers a rare low-level, step-by-step walkthrough of how large language model inference actually runs on GPU hardware, which could serve as an educational resource for engineers wanting to understand transformer internals beyond high-level frameworks. Because the author emphasizes writing the notes without AI assistance, it also positions the material as a human-authored reference at a time when much technical content is increasingly AI-generated.
- The engine is built entirely in C++17 and CUDA, loading Llama 3.2 1B Instruct weights directly from safetensors files.
- It implements a full forward pass including tokenization, embedding, RMSNorm, attention, and MLP layers across all 16 transformer layers.
- Documentation includes 6+ hours of video and an 85-page handwritten PDF of notes, explicitly written without AI assistance.
NVIDIA GeForce RTX 4070 Super GPU — If you're building a CUDA-based LLM inference engine from scratch, you'll want a capable NVIDIA GPU to actually run and profile your kernels locally. The RTX 4070 Super offers solid CUDA core counts and VRAM for experimenting with transformer workloads like loading BF16 weights and running attention layers. It's a great companion for hands-on learning projects like this inference-from-scratch series.
See NVIDIA GeForce RTX 4070 Super GPU on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: github.com, 2026-11-09
Published there as: “Hours of YouTube and 85 handtyped pages of notes on LLM Inference – Start here”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.