Google has joined Anthropic, OpenAI and Microsoft in committing to recognize a proposed robots.txt directive that lets publishers block AI training crawlers with a single common signal instead of separate rules per company. The proposal, authored by Anthropic's Kevin Kelley and under consideration at the IETF, introduces a dedicated user-agent tag that can block or selectively allow scraping for AI training without affecting normal search indexing. Anthropic has already begun honoring it, Microsoft and Google plan support in 2027, and OpenAI says it will implement it within months.
engadget.com
· 2026-10-31
The four AI companies say they will support a new 'policy token' signal, proposed by Anthropic's Kevin Kelley through the Internet Engineering Task Force, that lets publishers set AI training preferences once within their robots.txt file rather than contacting each company separately. Anthropic has already adopted the signal, OpenAI says it will implement it, and Google and Microsoft plan to support it next year, with Microsoft also offering separate controls for AI training versus AI grounding.
theverge.com
· 2026-10-30
Cloudflare announced it will offer an OHTTP Gateway as a paid add-on for customers' zones, allowing businesses to receive Oblivious HTTP traffic without seeing users' IP addresses. The product complements Cloudflare's existing Privacy Gateway relay service, launched in 2022, and is open for waitlist registration ahead of its fall release.
blog.cloudflare.com
· 2026-10-03
GoKawiil