GoKawiilWandering Thoughts blog begins blocking visitors with generic HTTP User-Agent strings
Chris Siebenmann, who runs the Wandering Thoughts blog and CSpace wiki, says he is now attempting to block requests from clients whose HTTP User-Agent header is too generic or suspicious. He attributes the move to a surge in high-volume crawlers, some reportedly gathering data for LLM training, that strain his site's resources.
GoKawiil's interpretation of the reporting above, not reported fact.
The policy illustrates how individual site operators are responding to aggressive AI-related web scraping by tightening technical requirements rather than relying on robots.txt alone. Siebenmann's specific rules—requiring real identifying URLs for bots or 'compatible' user agents—could serve as a template other small sites adopt, potentially making it harder for poorly configured crawlers and legitimate tools alike to access independently run content.
- Site operator Chris Siebenmann is blocking visitors with overly generic User-Agent headers on his blog and wiki.
- He says a rise in high-volume crawlers, some for LLM training, prompted the change.
- His rules require bots or 'compatible' clients to include a real, working URL identifying themselves, not just an email address.
Source: utcc.utoronto.ca, 2026-10-14
Published there as: “Compiler bootstrapping depends on more than you might think”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.