Audit of 40 sites finds gaps in AI-readiness files across llms.txt, robots.txt, Shopify endpoints
A 40-domain study examined how sites prepare for AI crawlers like GPTBot and Perplexity, checking structured data, llms.txt files, robots.txt rules, and Shopify's MCP commerce endpoints. It found only 7.5% of homepages include FAQPage structured data, many llms.txt files are boilerplate or soft-404s, no domain blocks GPTBot or PerplexityBot via robots.txt, and 80% of D2C Shopify stores expose /api/ucp/mcp endpoints while none have customized /agents.md files.
GoKawiil's interpretation of the reporting above, not reported fact.
The findings suggest many brands are technically open to AI agents and assistants without having curated what those systems actually see, which could mean AI-generated answers about pricing, returns, or service terms rely on outdated or generic data rather than brand-specific information. The contrast between developer SaaS (100% genuine curation) and D2C brands (10%) hints that preparedness for AI search and agentic commerce varies sharply by industry, a gap the study's author frames as a competitive risk for laggards.
- Only 7.5% of sampled homepages provide FAQPage structured data for AI assistants to parse accurately.
- None of the 40 domains block GPTBot or PerplexityBot in robots.txt, though two return 403s at the CDN level.
- 80% of D2C Shopify stores expose commerce API endpoints but 0% have customized agent-facing configuration files.
Source: verisource.dev, 2026-11-17
Published there as: “40-domain study: 80% of Shopify stores expose /API/ucp/MCP, 0% /agents.md”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.