The foundational deal of the modern web, a handshake agreement that powered two decades of search and content, is officially dead. Cloudflare just put a price on scraping the internet, and it’s coming for artificial intelligence’s free lunch.
Almost 30 years ago, two Stanford grad students, Larry Page and Sergey Brin, built Google on a simple bargain: content creators would let them copy the entire web in exchange for traffic. For years, that traffic powered ad revenue, subscriptions, and the growth of online media. Google mostly upheld its end of the deal. But that era is collapsing under the weight of AI.
On July 1, Cloudflare, one of the internet’s core infrastructure companies, declared “Content Independence Day.” In a landmark policy shift, the company announced it will now block AI crawlers from scraping sites hosted on its platform unless those bots pay content creators for the data they consume.
“Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content,” CEO Matthew Prince announced in a blog post. “That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”
This is a sharp, aggressive turn from the web’s traditionally open access ethos. Cloudflare argues it’s long overdue. AI tools like ChatGPT and Google’s own AI Overviews are now answering user questions directly, effectively strip-mining websites for information while sending almost no traffic back to the original source.
“Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value,” Prince said.
Publishers, we see you! 🙌 Cloudflare just launched pay per crawl to put control over your content back where it belongs. Now, crawling is more transparent and controlled, by default, creating a better web ecosystem for creators like you. This is about real content… pic.twitter.com/yatB5LSBIm — Cloudflare (@Cloudflare) July 2, 2025
The Web as Swiss Cheese
The numbers are stark. Cloudflare claims it’s already 10 times harder to get traffic from Google than it was a decade ago due to features like the answer box. But the new AI models are far worse. According to Cloudflare’s internal metrics, OpenAI drives 750 times less traffic than traditional Google search, while Anthropic drives a staggering 30,000 times less. The reason is simple: people are asking ChatGPT instead of Googling. The content still gets used, but the creators have been completely cut out of the value chain.
Using its position as a gatekeeper for roughly 20% of all websites globally (around one-fifth of all web traffic passes through Cloudflare’s network), Cloudflare is now forcing the issue by charging a toll.
... continue reading