A high-stakes war has just broken out over the future of the internet. In one corner is Cloudflare, a giant of web infrastructure that acts as a gatekeeper for a huge portion of online traffic. In the other is Perplexity, a darling of the AI world, a search engine threatening to upend Google’s dominance.
The accusation is explosive: Cloudflare claims Perplexity is a bad actor, a rogue bot that ignores the internet’s oldest rules to secretly scrape data from websites that have explicitly told it to stay away. Perplexity’s response is just as fiery: it says Cloudflare is either dangerously incompetent or engaged in a publicity stunt, fundamentally misunderstanding how modern AI works.
The feud is the first major battle in a conflict that will define the next era of the web: Who gets to access online information, and who gets to decide the rules?
The Accusation: A Rogue Bot in Disguise
For decades, the internet has operated on a “gentleman’s agreement” called the robots.txt file. It’s a simple text file that website owners use to post a digital “Do Not Enter” sign for automated web crawlers or “bots.” Well-behaved bots, like Google’s, respect this sign.
In a scathing blog post, Cloudflare alleges that Perplexity is ignoring it. The company claims that when its declared bot, “PerplexityBot,” is blocked, the AI search engine switches to stealth mode, using generic browser identities and rotating IP addresses to continue crawling and gathering data in disguise.
Cloudflare says it tested this by creating brand-new, private websites with strict “no bots allowed” rules. Despite this, they found that “Perplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains.” Based on this “stealth crawling behavior,” Cloudflare announced it has now de-listed Perplexity as a verified bot and is actively blocking its undeclared crawlers.
The Rebuttal: “You Don’t Understand How AI Works”
Perplexity’s response was swift, accusing Cloudflare of getting “almost everything wrong about how modern AI assistants actually work.” The company argues that it is not a traditional “bot” and that Cloudflare is misapplying old rules to new technology.
The core of their argument is the difference between a bot and a user agent. A traditional bot, like Google’s, systematically crawls billions of pages to build a massive index for later use. A user agent, Perplexity claims, acts on behalf of a real person in real-time. When you ask Perplexity a question, its AI agent fetches the necessary information from the web at that moment to answer you. It’s not stockpiling data; it’s acting as your personal research assistant.
... continue reading