GoKawiilThese days, the frontier AI labs are all racing to build self-improving models. Some believe it’s the surest route to superintelligence—as AI improves itself in a mind-melting loop, the thinking goes, it will eventually surpass human comprehension (and perhaps even control).
That’s all well and good, but I have a newsletter to produce. I wondered if recursive self-improvement might also be useful for me. Could I use AI to train and continually improve a model that automates some of this newsletter’s busywork?
After a week or so of experimenting, the answer appears to be a resounding—and surprising—hell yes. What’s more, dabbling with self-improving models shows a different vision for how AI might unfold—one that doesn’t center on a handful of companies that control the whole industry.
I started by trying out a simple self-improving loop
To get my feet wet, I experimented with training a small language model from scratch—by which I mean I dumped all the hard work on Claude’s plate.
I installed AutoResearch, which helps an off-the-shelf AI model build and improve a smaller model. AutoResearch is the brainchild of Andrej Karpathy, a superstar AI researcher who helped found OpenAI, led AI work at Tesla, and recently joined Anthropic.
I fired up Claude and gave it the recommended instruction: “Hi, have a look at program.md and let's kick off a new experiment!” While Claude did the hard stuff, I provided silicon (an Nvidia DGX, a desktop “supercomputer” designed for AI experimentation), the electricity (running hot for a few days straight), and a possibly ill-advised willingness to let the model skip all the usual permission checks in order to do its thing (let him cook!)
I checked in on the AutoResearch project every few hours and marveled as Claude adjusted parameters and training regimes, looked at how this changed the smaller model’s output, and went on refining it further.
Here’s what an early version of that smaller language model produced when I prompted it to complete the phrase “In the beginning …”
“In the beginning of the beginning of the end of the end of the end end of end end end end end end end end beginning end end end end…”
... continue reading