GoKawiilStudy of 217M Common Crawl pages finds 7,959 sites sending AI-targeted HTTP headers
Researchers scanned 217 million HTTP response headers from a 10% sample of Common Crawl's September 2026 crawl and found 7,959 sites including headers addressed to AI agents. Most, 7,271 sites, were pointers directing agents to resources like llms.txt files or MCP servers, while 30 sites contained attempted prompt injections and 139 sites flagged whether they had identified the visitor as a bot.