GoKawiiltl;dr - I trained a tiny Mamba-based autoregressive language model and wrote an inference engine to run it on the 8-bit 6502 processor (from 1975, with 32KB RAM). Running it on my dad's BBC Micro generated the text below.
once upon a time tom and lily saw things lily were sad her house he heartd them ilily and tom said yes she saw a little girl smiled tom was so excited her mom said yes
The MOS 6502 is an 8-bit microprocessor released in 1975, powering the BBC Micro and the Apple II. I am lucky enough to have access to my dad's BBC Model B from the 80s; I wanted to see, using modern machine learning, what the strongest language model we could fit on this machine was.
Unsurprisingly, this poses significant challenges. The model weights and inference code need to be contained within 25KB of user-space memory — my final configuration was 9KB inference code and 13KB model weights. The CPU only operates on an 8-bit integer datatype, and doesn't include multiplication in its instruction set.
CC65 is used for the inference code, enabling compilation of C to the 6502 instruction set. A binary of a model trained on my MacBook can then be written to the BBC Micro using PlayUEF and a custom 3.5mm-to-tape cable I DIY'ed. This convinces the BBC that it's listening to a tape drive, while my laptop plays audio out of its headphone jack.
The sim65 emulator allows a parity check between the C inference binary and the reference Python model implementation. The full inference engine can be tested on the jsbeeb emulator before running on the BBC Micro.
You can run it yourself — the link below boots a BBC Micro in your browser, loads the UEF tape image straight from GitHub, and auto-types the commands to run the model. No emulator install required (note that generation takes a few minutes).
Run BitNet on a BBC Micro →
Modelling
The goal of this project is to build an autoregressive language model — a language model that produces tokens one-by-one, similar to frontier language models. The model is a function $f$ that produces the next token from the existing context:
... continue reading