iStock / Getty Images Plus
The major internet Content Delivery Network (CDN), Cloudflare, has declared war on AI companies. Starting July 1, Cloudflare now blocks by default AI web crawlers accessing content from your websites without permission or compensation.
The change addresses a real problem. My own small site, where I track all my stories, Practical Technology, has been slowed dramatically at times by AI crawlers. It's not just me. Numerous website owners have reported that AI crawlers, such as OpenAI's GPTBot and Anthropic's ClaudeBot, generate massive volumes of automated requests that clog up websites so they're as slow as sludge. GoogleBot alone reports that the cloud-hosting service Vercel bombards the sites it hosts with over 4.5 billion requests a month.
These AI bots often crawl sites far more aggressively than traditional search engine crawlers. They sometimes revisit the same pages every few hours or even hit sites with hundreds of requests per second. While the AI companies deny that their bots are to blame, the evidence tells a different story.
Also: Senate removes ban on state AI regulations from Trump's tax bill
Thus, on behalf of its two million-plus customers, 20% of the web, Cloudflare now blocks AI crawlers. For any new website signing up for its services, AI crawlers will be automatically blocked from accessing its content unless the site owner grants explicit permission. Additionally, Cloudflare promises to detect "shadow" scrapers — bots that attempt to evade detection — by using behavioral analysis and machine learning. What's good for the AI goose is good for the gander.
This move reverses the previous status quo, where website owners had to opt out of AI crawling. Now, blocking is the default, and AI vendors must request access and clarify their intentions, whether for model training, search, or other uses, before they're allowed in.
This change arises not only because of frustrated website owners. Numerous publishing companies, such as The Associated Press, Condé Nast, and ZDNET's own parent company, Ziff Davis, are frustrated that AI companies have been "strip mining" the web for content. All too often, this has been done without compensation or consent, and sometimes, ignoring standard protocols like robots.txt that are meant to block crawlers.
(Disclosure: Ziff Davis, ZDNET's parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
Moreover, recent court cases have ruled in favor of Meta and Anthropic, finding that their use of copyrighted works was legal under the doctrine of fair use. Needless to say, writers, artists, and publishers don't like this one bit. Publishers are still worried that the federal government will give AI free rein to do as it wants with their content. AI powerhouses such as OpenAI and Google are continuing to lobby the government to classify AI training on copyrighted data as fair use.
... continue reading