GoKawiilGoogle, OpenAI, Anthropic, Microsoft to honor new AI-training robots.txt signal
Google has joined Anthropic, OpenAI and Microsoft in committing to recognize a proposed robots.txt directive that lets publishers block AI training crawlers with a single common signal instead of separate rules per company. The proposal, authored by Anthropic's Kevin Kelley and under consideration at the IETF, introduces a dedicated user-agent tag that can block or selectively allow scraping for AI training without affecting normal search indexing. Anthropic has already begun honoring it, Microsoft and Google plan support in 2027, and OpenAI says it will implement it within months.
GoKawiil's interpretation of the reporting above, not reported fact.
The move could simplify how publishers control AI data harvesting, replacing a patchwork of crawler-specific rules with one standardized opt-out mechanism. However, since the directive only affects future crawling and not data already collected, publishers may see limited immediate relief, and the long rollout timelines from Microsoft and Google suggest the industry is not treating this as urgent.
- New robots.txt tag lets publishers block AI training crawlers with one signal instead of many
- Anthropic's Kevin Kelley authored the proposal, now under IETF review
- Directive won't remove already-scraped data and companies vary widely in rollout timing, from immediate to 2027
Source: engadget.com — Mariella Moon, 2026-10-31
Published there as: “Google joins OpenAI and Anthropic in promising new scraping controls”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.