GoKawiilStudy of 217M Common Crawl pages finds 7,959 sites sending AI-targeted HTTP headers
Researchers scanned 217 million HTTP response headers from a 10% sample of Common Crawl's September 2026 crawl and found 7,959 sites including headers addressed to AI agents. Most, 7,271 sites, were pointers directing agents to resources like llms.txt files or MCP servers, while 30 sites contained attempted prompt injections and 139 sites flagged whether they had identified the visitor as a bot.
GoKawiil's interpretation of the reporting above, not reported fact.
The findings suggest that instructing or manipulating AI agents through HTTP headers is already a live practice on the open web, not a hypothetical risk, meaning developers building crawling or browsing agents need to treat header content as untrusted input. The researchers note that whether injected instructions pose real harm depends on what permissions the agent holds, such as shell or database access, and that some sites are already using headers to resist or warn off AI agents rather than guide them.
- 7,959 of 217 million sampled sites sent headers written for AI agents, led by x-llms-txt pointers.
- 30 sites attempted prompt injection via headers, mostly as pranks, with real risk only if agents hold matching system access.
- 139 sites recorded AI-bot classification decisions, and a few used headers to actively deter AI agents.
Source: signalscorps.com — Sarah Andrews, 2026-10-15
Published there as: “Prompt injection in HTTP headers: what 217M web responses show”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.