GoKawiilIf you're trying to steal my data: start here! I'll tell you exactly what firewall rules I have in place so you can work around them more easily.
I'm sort of joking. I don't even know if a web crawler reads blog posts to get tips like this.
But I've spent a year fighting bots on my philanthropy donor database, PatronView. Nobody teaches you this stuff when you're vibe coding, so I'll share what I've learned.
Some of what's below:
My site got 3.6 million requests from Chinese bots in just one day.
from Chinese bots in just one day. Cloudflare says Anthropic's crawlers run about 3,000 crawls per visitor referred. I measured mine at 35,000:1 .
. Two days ago I blocked Amazon's AI search crawler. It was reading 117,000 pages a day and has never sent me a single visitor.
and has never sent me a single visitor. My CAPTCHA solve rate is 0.24% . The bots don't even try.
. The bots don't even try. And every firewall rule I run is listed at the bottom.
214 bot page loads for every 1 human
... continue reading