Skip to content
Tech News
← Back to articles

When su replaced login for becoming another Unix login

read original more articles
Why This Matters

This article highlights the importance of clear and specific User-Agent headers in HTTP requests, especially as high-volume crawlers threaten server resources. It underscores the need for transparency and accountability in automated web traffic, which benefits both industry operators and consumers by reducing server load and improving security.

Key Takeaways

You're using a tool with a too-generic User-Agent

You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace, the wiki thing it's part of. Unfortunately whatever you're using to do so has a HTTP User-Agent header value that is too generic or otherwise excessively suspicious. Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that behave like this. To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this.

All HTTP User-Agent headers should clearly identify what they are, and for non-browser user agents, they should identify not just the software involved but also who specifically is using that software. An extremely generic value such as " Go-http-client/1.1 " is not something that I consider acceptable any more.

One specific thing is that if your HTTP User-Agent header contains an explicit marker that you are only 'compatible' with what your User-Agent suggests, you must include an explicit, real URL that describes your activities or the software. A minimal HTTP User-Agent such as 'Mozilla/5.0 (compatible; mysoftware)' is not acceptable.

Chris Siebenmann, 2026-04-18