Skip to content
Tech News
← Back to articles

LLM scraper bots are overloading acme.com's HTTPS server

read original get Web Scraper Bot Detection → more articles
Why This Matters

This incident highlights how aggressive scraping by large language model (LLM) bots can overload web servers, causing outages and degraded performance. It underscores the importance for website operators to implement effective traffic management and security measures to maintain service stability amid increasing automated bot activity.

Key Takeaways

ACME Updates

07Apr2026 https outage

Starting on February 25th and lasting a little over a month, acme.com suffered intermittent network outages. The symptoms were very high ping times and packet drops. The outages would last a few hours and then go away for a while.

The problems started right after my internet provider, Sonic, did some scheduled maintenance and switched me to a new network. I worked with Sonic support to try and figure out if their network change, or my own config changes for the new network, could have caused the problems. I did find some things I did wrong, but fixing them didn't fix the outages.

A few days ago I was up at 1am, filled with anxiety at yet another outage, and decided to take a harder look at the traffic I was getting. I noticed some interesting things:

Nearly all the incoming packets were web requests.

Nearly all of them were for non-existent pages.

Nearly all of them were on port 443 / https, not port 80 / http.

Nearly all of them proudly announced in the user-agent that they were LLM scraper bots.

I run two web servers on ACME, one for http that is very fast, and another for https which is kind of slow. Maybe the slow https server was falling behind? I decided to test this: I closed port 443.

... continue reading