GoKawiilOpenAI, Google, Microsoft and Anthropic back shared opt-out standard for AI scraping
The four AI companies say they will support a new 'policy token' signal, proposed by Anthropic's Kevin Kelley through the Internet Engineering Task Force, that lets publishers set AI training preferences once within their robots.txt file rather than contacting each company separately. Anthropic has already adopted the signal, OpenAI says it will implement it, and Google and Microsoft plan to support it next year, with Microsoft also offering separate controls for AI training versus AI grounding.
GoKawiil's interpretation of the reporting above, not reported fact.
A shared standard could simplify how publishers protect their content from unauthorized scraping, replacing a patchwork of company-specific rules with one universal signal. It follows other industry efforts like the Really Simple Licensing standard and Cloudflare's default AI-crawler blocking, suggesting growing pressure on AI firms to address publisher concerns, though adoption timelines vary and enforcement depends on each company's compliance.
- A new 'policy token' lets publishers opt out of AI scraping once via robots.txt, instead of per company.
- Anthropic has adopted it already; OpenAI plans to, while Google and Microsoft will support it next year.
- Microsoft will offer separate opt-outs for AI training and AI grounding, but sites still appear in Bing search regardless.
Source: theverge.com — Emma Roth, 2026-10-30
Published there as: “The biggest names in AI are backing a new way to opt out of scraping”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.