GoKawiilWhy we rewrote Antfly's Go engine in Zig while the startup was still early: first principles, caring about the model instead of the embeddings, TigerBeetle-style simulation testing, and what the rewrite made possible.
My colleague Rowan summarized our ambitious goal for Antfly over a year ago: perfect search! This goal is silly, ambitious, unobtainable, and perfect for us. Hearing grandiose technologists talk about perfect search still strikes me as something out of an episode of Silicon Valley, but I love an impossible goal, and my inner tech hipster appreciates the irony. Aim for perfect search and you miss, but you miss somewhere interesting. So I'll walk you through why we did the thing you are never supposed to do with a new startup: we rewrote the product. Antfly v0.1 launched a document storage, full-text, vector, and graph indexing engine in Go. Antfly v0.2 launches the same engine in Zig, with zero dependencies (well, none that get to run the show, but more on that below).
First principles #
I don't hear people talking about first principles as often as I used to, but they're still important to me and to how I make decisions. When building the first version of Antfly, the idea was to fill a gap in the database market: a schema-ish, friendly query engine for indexing like Elasticsearch, closer in scale to Postgres than Iceberg, as easy to use as Mongo, and as easy to operate as Google's Spanner or Bigtable. (And I thought perfect search was too lofty... HA!)
When building the first version I didn't have the same sort of software tools (cough cough Codex, Claude, Aider, Pi) available to assist in development, and the hard problems I wanted to tackle were the ones CockroachDB had similarly chosen Go for: distributed systems and concurrency. Rust can guarantee memory safety through lifetimes, but memory safety wasn't the hard part, and Go is infinitely more readable to me than Rust ever was. Plus, Go had the most battle-tested Raft implementation out there (etcd's), and I was only one person, working on weekends and on my Fridays, trying to build an Elasticsearch DSL on top of Bleve as well.
Caring about the model, not the embeddings #
Fast forward a little bit and embeddings started to become a thing. word2vec had shown that a vector could actually carry meaning, and the first practical embedding models meant an average dev could build Google-lite semantic search for their app. At work, our scale of vector storage was so small that a top-k could be an exhaustive search, but I read about all the fun algorithms behind Pinecone, Vertex AI Vector Search, Elastic, Mongo, CockroachDB, and pgvector after an old coworker rolled his eyes at the ridiculous Pinecone seed round. Pinecone might have been overvalued, but I think one of its most interesting ideas was overlooked: users could care about the model instead of having to care about the embeddings. It made me think about Postgres, and how for the most part a user can avoid knowing about B-trees and other indexing algorithms, or how in Bleve and Lucene you can avoid knowing about S2 indexes, finite state transducers, and so on. Vector databases and indexes, on the other hand, required you to know and care about HNSW, SPFresh, RaBitQ, and the rest.
I decided to try my hand at implementing these algorithms from their papers and blog posts in the initial version of Antfly, and it was wildly successful. I was able to make a semantically searchable Wikipedia using my laptop, Antfly, and Ollama! Embedding generation was so slow that the database being a little bit slower was not a big concern for the initial implementation.
Enter Zig #
Concurrently with all this, I had tried my hand at implementing VSR, Protobuf, and an LSM in Zig a few years prior, after stumbling upon TigerBeetle. I really liked the readability of Zig, the concurrency primitives, the people implementing the language, and the interoperability with C. But Zig was too green for my weekend database project and lacked a lot of the heavy lifting: Raft, full-text indexing, a portable, battle-tested LSM, yada yada yada. I took a lot of the spirit of the TigerBeetle folks with me, though, and put myself to work incorporating VOPR testing with TLA+ trace validation (Rowan's post on formal verification with coding agents covers that), built on the new Go mock time and on prior art from etcd's Raft TLA+ spec and trace validation.
... continue reading