GoKawiilCloudflare adds setting to block AI training without losing search indexing
Cloudflare launched a 'Disallow AI Training' control that lets website owners stay visible in search results while blocking the same crawler from using their content to train AI models. Apple, Google and Microsoft have agreed to honor the new setting, which addresses the problem of mixed-use crawlers that previously forced sites to accept both search indexing and AI training together or neither.
GoKawiil's interpretation of the reporting above, not reported fact.
Publishers relying on ad and subscription revenue have had no way to separate search visibility from AI training consent, since major crawlers serve both purposes at once. Cloudflare data shows almost no sites want to block search, but 17% try to block training, revealing pent-up demand for finer control that a simple robots.txt file cannot enforce. Cloudflare is also planning similar granular controls for AI-generated summaries by early next year, which could reshape how content owners negotiate with AI companies.
- Cloudflare's new setting decouples search indexing from AI training permissions for mixed-use crawlers.
- Apple, Google and Microsoft have committed to respecting the opt-out.
- Cloudflare plans to extend similar granular controls to AI summaries by early next year.
Source: blog.cloudflare.com, 2026-09-16
Published there as: “Stay discoverable in search while disallowing AI training”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.