GoKawiilThis text was written with the assistance of an AI. Careful: assistance doesn't mean the AI wrote it. It means it corrected, reviewed and filled in some parts, but the author is human (or so I believe).
Léeme en español
Qapla' Project
This is how you train a transformer from scratch on an eight-buck ESP32-S3.
The (Klingon) GPT nobody asked for but everybody needed.
And why did we need it? Because we take for granted that training a model requires a GPU or a datacenter. Not always: sometimes something as small and as cheap as an eight-buck micro is enough to train one from scratch.
Read that again: train. From scratch. Not run a pre-cooked model. Train. Forward pass, backprop and weight updates, inside the chip.
What is Qapla'?
Edge AI is nothing new. TinyML has been doing inference on microcontrollers for years, community ports of llama2.c — Andrej Karpathy's minimalist project — put ~260K-parameter transformers on an ESP32, and just recently a brilliant project got a model of almost 29 million parameters running on an $8 ESP32-S3. They're excellent pieces of work, and although they happen to have overlapped in time with this experiment, they weren't the model we were following.
The esp32-ai project by Slava S. (slvDev), and others like it, all share one thing: they are inference. The model is born somewhere else — a GPU, a datacenter — trained on its data, quantized, and only then loaded onto the chip so it can run it. The brain is cooked outside and served on-chip.
... continue reading