You should feed the bots:
Aug 3, 2025
A week ago, I set up an infinite nonsense crawler trap – now it makes up 99% of my server’s traffic. What surprised me is that feeding scrapers garbage is the cheapest and easiest thing I could do.
Meet the bots:
These aren’t the indexing bots of old, but scrapers collecting data to train LLMs. Unlike search engines, which need the websites they crawl to stay up, AI companies provide a replacement.
It should come as no surprise that these bots are aggressive and relentless: They ignore robots.txt, and if block them by user agent they just pretend to be a browser. If you ban their IP, they switch addresses.
… all while sending multiple requests per second, all day, every day.
Giving up:
So what if we let them access the site?
Serving static files is is relatively cheap, but not free. SSD access times are in the tens milliseconds, and that’s before you pay the filesystem tax. Bots also like to grab old and obscure pages, ones that are unlikely to be in cache. As a result, it doesn’t take all that many requests to bog down the server.
... continue reading