GoKawiilHow I built a 500k-Domain Search Engine for Makers in a Weekend for $10
Sunday, 2am. I couldn’t sleep and I was annoyed at search engines again. Every query I actually cared about, portfolios, zines, weird little art projects, one-person software, drowned under a foot of corporate documentation and SEO sludge. So I did the thing you do at 2am: I opened a terminal and typed out a plan.
“I want to make a search engine for myself only. There are 40-ish million domains. We can store a bit of metadata about each one. Even with 1KB each that’s 40GB, which is doable.”
By Wednesday lunch I had 560,183 homepages catalogued, an empty queue of anything worth fetching next, and a decision to stop. This is the story of that weekend: what I built, what broke, what it cost, and what I’d tell you if you wanted to build your own.
The headline, if you only read one paragraph: for about $10, an overnight GPU rental, and a few hours of steering the thing while it ran, you can have a personal search index of a few hundred thousand sites, under a gigabyte on disk. That’s the whole pitch. Everything below is how I got there and where the sharp edges are.
Full technical details are provided separately.
What I was actually trying to build
Not “index the web.” Just: find people doing stuff, art, code, hardware, poetry, little theatres, and not drown in docs.company.com . Personal, single user, no accounts. A crawler that only ever looks at homepages, a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. A little search UI on top with fuzzy matching so I could type half a word and still find the right site.
I wrote down what I was explicitly not building, mostly so that agents helping me wouldn’t quietly “simplify” it into something bigger: no IP scanning, no Redis, no storing full page HTML, no recrawl scheduler, nothing multi-tenant. Ignore was just a checkbox on a category, applied at search time. The crawler still summarised ecommerce sites, I just didn’t have to look at them.
The napkin math was tens of millions of domains at roughly 1KB of metadata each, which is genuinely nothing for a Postgres box. Page text itself was never meant to be a corpus, just a scratch buffer that gets thrown away the moment the model is done with it.
... continue reading