Elyse Betters Picaro / ZDNET
ZDNET's key takeaways
Cloudflare claims Perplexity ignores websites' wishes in its content hunt.
Other AI companies, such as OpenAI, don't wipe content, Cloudflare says
Cloudflare now offers services to block aggressive AI crawlers.
Perplexity is denying Cloudflare's claims.
Cloudflare, a leading content delivery network (CDN) company, has accused the AI startup Perplexity of evading websites' "no crawl" directives by stealthily deploying web crawlers to scrape content from sites that have explicitly blocked its official bots.
If that sounds familiar, you've heard these accusations before. Last year, WIRED and Forbes both accused Perplexity of doing the same thing to their sites.
How Perplexity allegedly bypasses 'no crawl' directives
According to Cloudflare, when Perplexity's web crawler encounters a robots.txt file, which sites use to block their content from being crawled, Perplexity pretends to be an ordinary Chrome web browser on a Mac. This enables it to bypass the bot barriers.
... continue reading