Blocking LLMs from your website cuts you off from next-generation search
Why blocking LLMs from your website is dumb John Wang 2 min read · 1 hour ago 1 hour ago -- Listen Share Perplexity was recently accused of scraping sites that had explicitly disallowed LLM crawlers in their robots.txt files. In the wake of that revelation, a wave of how-to guides for blocking large-language-model scraping has surfaced [0]. They’re generally highly vitriolic, with people opposing this on both moral grounds (“AI is stealing your content”) as well as displaying a general distaste