A valid HTML zip bomb
Many sites have been affected by the aggressiveness of web crawlers designed to improve LLMs. I’ve been relatively spared, but since the phenomenon started, I've been looking for a solution to implement. Today, I present a zip bomb gzip and brotli that is valid HTML. The initial problem is the aggressiveness of LLM web crawlers that don't respect robots.txt . The first idea that comes to mind is IP blocking. However, web crawlers have circumvented this restriction by using individual IPs via