GoKawiilThe Agentic Web Index The internet is rapidly evolving from an environment built primarily for humans, into one increasingly used by machines. See how AI agents, crawlers, scrapers, and other bots are reshaping the way information is discovered, accessed, and used across the web.
Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking.
Traffic by Agent Type AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user AI Agent AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer AI Assistant AI Coding Agent Fetches documentation and other resources to help build software AI Coding Agent AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service AI Data Provider AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs AI Data Scraper AI Search Crawler Indexes website content to possibly include as citations in AI-powered search results AI Search Crawler Archiver Captures and stores historical website snapshots for long-term digital preservation Archiver Automated Agent Automates browser interactions programmatically without direct human supervision Automated Agent Developer Helper Assists with testing, debugging, and ensuring website functionality Developer Helper Fetcher Retrieves web page metadata to power app features like link previews or feeds Fetcher Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting Intelligence Gatherer Scraper Extracts large amounts of web data, often without explicit website permission Scraper Search Engine Crawler Systematically scans and indexes web pages to include in search results Search Engine Crawler Security Scanner Scans websites for security vulnerabilities, threats, and configuration weaknesses Security Scanner SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities SEO Crawler Uncategorized Not yet assigned a type Uncategorized Undocumented AI Agent Crawls websites without disclosing its purpose, collecting data for an unknown AI use case Undocumented AI Agent
These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Providers and AI Data Scrapers.
Traffic by Agent Type AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service AI Data Provider AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs AI Data Scraper Top Visited Website Categories Law and Government 6.0% Online Communities 5.9% Home and Garden 5.5% Computers and Electronics 5.3% Shopping 5.2% Jobs and Education 5.2% Pets and Animals 5.1% Business and Industrial 5.0% Autos and Vehicles 4.8% Games 4.8% Hobbies and Leisure 4.6% Internet and Telecom 4.5% Beauty and Fitness 4.1%
Top Agents ClaudeBot AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 27.0% meta-externalagent AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 19.5% Amazonbot AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 19.2% GPTBot AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 9.4% Bytespider AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 7.3% ShapBot AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service PVDR 5.6% GoogleOther AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 3.9% CCBot AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 2.3% Timpibot AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 1.8% YouBot AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service PVDR 1.4% VelenPublicWebCrawler AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 0.8% Diffbot AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service PVDR 0.5% DeepSeekBot AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs SCRP 0.4% TerraCotta AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service PVDR 0.3% AIWebIndex AI Data Provider Crawls websites to supply structured content to AI systems as a third-party service PVDR 0.2% Top Operators Anthropic 27.0% Meta 19.5% Amazon 19.2% OpenAI 9.4% ByteDance 7.3% Parallel 5.6% Google 3.9% Common Crawl 2.3% Timpi 1.8% You.com 1.4% Velen 0.8% Diffbot 0.5% DeepSeek 0.4% Ceramic 0.3% Lyrenth 0.2%
These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Assistants and AI Coding Agents.
Traffic by Agent Type AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer AI Assistant AI Coding Agent Fetches documentation and other resources to help build software AI Coding Agent Top Visited Website Categories Reference 4.0% Science 3.7% Finance 3.4% Law and Government 2.6% Computers and Electronics 2.3% Beauty and Fitness 2.1% Travel and Transportation 1.9% Business and Industrial 1.8% Internet and Telecom 1.7% Health 1.6% Autos and Vehicles 1.3% Sports 1.3% Real Estate 1.3%
Top Agents ChatGPT-User AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 86.4% DuckAssistBot AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 3.3% Perplexity-User AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 2.7% Claude-User AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 2.5% MistralAI-User AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 2.1% Claude-Code AI Coding Agent Fetches documentation and other resources to help build software CODE 1.1% Shap-User AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 1.0% Google-NotebookLM AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 0.4% Cursor AI Coding Agent Fetches documentation and other resources to help build software CODE 0.4% Gemini-Deep-Research AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 0.0% meta-externalfetcher AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 0.0% GoogleAgent-URLContext AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 0.0% Code AI Coding Agent Fetches documentation and other resources to help build software CODE 0.0% kagi-fetcher AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 0.0% QualifiedBot AI Assistant Fetches website content in response to a user prompt, to include in an AI-generated answer ASST 0.0% Top Operators OpenAI 86.4% Anthropic 3.6% DuckDuckGo 3.3% Perplexity 2.7% Mistral 2.1% Parallel 1.0% Google 0.4% xAI 0.4% Meta 0.0% GitHub 0.0% Amazon 0.0% Kagi 0.0% Qualified 0.0% Poggio 0.0% ByteDance 0.0%
These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers.
... continue reading