GoKawiilGoogle, OpenAI, Anthropic, Microsoft to honor new AI-training robots.txt signal
Google has joined Anthropic, OpenAI and Microsoft in committing to recognize a proposed robots.txt directive that lets publishers block AI training crawlers with a single common signal instead of separate rules per company. The proposal, authored by Anthropic's Kevin Kelley and under consideration at the IETF, introduces a dedicated user-agent tag that can block or selectively allow scraping for AI training without affecting normal search indexing. Anthropic has already begun honoring it, Microsoft and Google plan support in 2027, and OpenAI says it will implement it within months.