Skip to content
Tech News
← Back to articles

Audit of 40 sites finds gaps in AI-readiness files across llms.txt, robots.txt, Shopify endpoints

read original more articles
GoKawiil Brief

A 40-domain study examined how sites prepare for AI crawlers like GPTBot and Perplexity, checking structured data, llms.txt files, robots.txt rules, and Shopify's MCP commerce endpoints. It found only 7.5% of homepages include FAQPage structured data, many llms.txt files are boilerplate or soft-404s, no domain blocks GPTBot or PerplexityBot via robots.txt, and 80% of D2C Shopify stores expose /api/ucp/mcp endpoints while none have customized /agents.md files.

Why It Matters

GoKawiil's interpretation of the reporting above, not reported fact.

The findings suggest many brands are technically open to AI agents and assistants without having curated what those systems actually see, which could mean AI-generated answers about pricing, returns, or service terms rely on outdated or generic data rather than brand-specific information. The contrast between developer SaaS (100% genuine curation) and D2C brands (10%) hints that preparedness for AI search and agentic commerce varies sharply by industry, a gap the study's author frames as a competitive risk for laggards.

Key Takeaways

Source: verisource.dev, 2026-11-17

Published there as: “40-domain study: 80% of Shopify stores expose /API/ucp/MCP, 0% /agents.md”

Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.
Keep reading
shopify 16 briefs → robots.txt 9 briefs → llms.txt 3 briefs →