Cloudflare experiment will block AI bot scrapers unless they pay a fee
Cloudflare has rolled out a couple of new measures meant to keep AI bot crawlers at bay. To start with, every new domain customer that signs up with the company to manage their website traffic will now be asked if they want to allow AI crawlers or to block them altogether. The company released a free tool in 2024 to block AI bots, but with this change, users can block them by default without having to tinker with their settings. Several big publishers, including Condé Nast, TIME and The Associat