Tech News
← Home  ·  All topics

Gptbot

1 GoKawiil brief on this topic · RSS for gptbot

Audit of 40 sites finds gaps in AI-readiness files across llms.txt, robots.txt, Shopify endpoints

A 40-domain study examined how sites prepare for AI crawlers like GPTBot and Perplexity, checking structured data, llms.txt files, robots.txt rules, and Shopify's MCP commerce endpoints. It found only 7.5% of homepages include FAQPage structured data, many llms.txt files are boilerplate or soft-404s, no domain blocks GPTBot or PerplexityBot via robots.txt, and 80% of D2C Shopify stores expose /api/ucp/mcp endpoints while none have customized /agents.md files.