GoKawiil
Tech News
Chris Siebenmann, who runs the Wandering Thoughts blog and CSpace wiki, says he is now attempting to block requests from clients whose HTTP User-Agent header is too generic or suspicious. He attributes the move to a surge in high-volume crawlers, some reportedly gathering data for LLM training, that strain his site's resources.
A blog operator built network and header-based rules on a Cloudflare Worker to sort incoming traffic without relying on JavaScript, and found that 277 of 372 requests bearing browser User-Agent strings were actually not genuine browsers—74.5% of the total. Comparing this edge-based counting with Cloudflare Web Analytics page loads over two days revealed persistent gaps between the two measurement methods, exposing bugs in the HTML-acceptance check and missing provenance markers, both since fixed.