A growing share of website traffic comes from AI crawlers. Once you know which crawler comes for what, you can decide deliberately who may read your content. This guide explains the differences and lists the currently known bots.
Three kinds of AI crawlers
AI bots serve different purposes. There are three types:
- Training/dataset crawlers collect content to train future models (e.g.
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended). Blocking protects content but lowers the chance of being "learned". - Search/index crawlers for answer engines build the index that AI answers cite from (e.g.
OAI-SearchBot,PerplexityBot). Blocking these directly costs AI visibility. - On-demand "user" fetches retrieve a page only when a user asks the AI about it (e.g.
ChatGPT-User,Perplexity-User). Behind such a fetch is a specific user question.
robots.txt and llms.txt: the difference
They work together but solve different jobs:
- robots.txt controls access: it tells a crawler which paths it may fetch. It is not content.
- llms.txt provides content: it offers your verified core knowledge in machine-readable form so AI systems understand and cite you correctly. → What is GEO?
Block or allow?
Most businesses want to be found and recommended. In that case, allow answer/search crawlers and on-demand fetches and also provide an llms.txt. You can block training crawlers selectively to protect content, but that also reduces how well models know your brand over time. Blocking everything across the board is a common mistake that costs visibility.
How to control access (robots.txt)
# Allow answer crawlers (recommended for visibility) User-agent: * Allow: / # Example: exclude one specific training crawler User-agent: GPTBot Disallow: / # Also point to your content source Sitemap: https://your-domain.com/sitemap.xml
A rule only applies to the exact user-agent it names. Since new bots keep appearing, a maintained reference list is useful.
Currently known AI crawlers (live)
We pull the following list live from a continuously maintained reference so it stays current:
Currently 188 AI crawler user-agents are documented. This list refreshes automatically as new bots appear:
AddSearchBot · AgentDataBot · AgentTimes · AI2Bot · AI2Bot-DeepResearchEval · Ai2Bot-Dolma · aiHitBot · AIWebIndex · amazon-kendra · amazon-QBusiness · Amazonbot · AmazonBuyForMe · Amzn-SearchBot · Amzn-User · Andibot · Anomura · anthropic-ai · ApifyBot · ApifyWebsiteContentCrawler · Applebot · Applebot-Extended · Aranet-SearchBot · atlassian-bot · Awario · AzureAI-SearchBot · bedrockbot · bigsur.ai · BixelBot · Bravebot · Brightbot · Brightbot 1.0 · BuddyBot · Bytespider · CCBot · Channel3Bot · ChatGLM-Spider · ChatGPT Agent · ChatGPT-User · Claude-Code · Claude-SearchBot · Claude-User · Claude-Web · ClaudeBot · Cloudflare-AutoRAG · CloudflareBrowserRenderingCrawler · CloudVertexBot · Code · cohere-ai · cohere-training-data-crawler · Cotoyogi · CragCrawler · Crawl4AI · Crawlspace · Cursor · Datenbank Crawler · DeepSeekBot · Devin · Diffbot · Diffbot-User · DoubaoBot · DuckAssistBot · Echobot Bot · EchoboxBot · ERNIEBot · ExaBot · ExaSearchBot · FacebookBot · facebookexternalhit · Factset_spyderbot · FirecrawlAgent · FriendlyCrawler · GeistHaus-PageFetcher · Gemini-Deep-Research · Google-Agent · Google-CloudVertexBot · Google-Extended · Google-Firebase · Google-Gemini-CLI · Google-NotebookLM · GoogleAgent-Mariner · GoogleAgent-URLContext · GoogleOther · GoogleOther-Image · GoogleOther-Video · GPTBot · GrokAppAndroid · GrokBot · HenkBot · Hunyuan · iAskBot · iaskspider · iaskspider/2.0 · ICC-Crawler · ImagesiftBot · imageSpider · img2dataset · ISSCyberRiskCrawler · kagi-fetcher · Kangaroo Bot · KeenableBot · Kimi-Agent · Kimi-SearchBot · Kimi-User · KimiBot · KlaviyoAIBot · KunatoCrawler · laion-huggingface-processor · LAIONDownloader · LCC · Lightpanda · LinerBot · Linguee Bot · LinkupBot · Manus-User · meta-externalagent · Meta-ExternalAgent · meta-externalfetcher · Meta-ExternalFetcher · meta-webindexer · MistralAI-Index · MistralAI-Training · MistralAI-User · MistralAI-User/1.0 · MoonshotBot · Mozilla-Tabstack · MyCentralAIScraperBot · NagetBot · nbot · netEstate Imprint Crawler · newsai · NotebookLM · NovaAct · OAI-AdsBot · OAI-SearchBot · omgili · omgilibot · OpenAI · opencode · Operator · PanguBot · Panscient · panscient.com · Perplexity-User · PerplexityBot · PetalBot · PhindBot · Poggio-Citations · Poseidon Research Crawler · qodercli · QualifiedBot · Querit-SearchBot · QueritBot · QuillBot · quillbot.com · QwenBot · Reflectionbot · SBIntuitionsBot · Scrapy · SemrushBot-OCOB · SemrushBot-SWA · Shap-User · ShapBot · Sidetrade indexer bot · Spider · TavilyBot · Terra Cotta · TerraCotta · Thinkbot · TikTokSpider · Timpibot · TongyiBot · Trae · TwinAgent · UseAI · VelenPublicWebCrawler · WARDBot · Webzio-Extended · webzio-extended · wpbot · WRTNBot · xAI-Grok · YaK · YandexAdditional · YandexAdditionalBot · YiBot · YiyanBot · YouBot · ZanistaBot
Source: the continuously maintained ai.robots.txt project, refreshed automatically · last change detected 17.10.2026.
Common mistakes
- Blocking all bots out of caution, then wondering about missing AI visibility.
- Mistaking robots.txt for a content solution (it governs access, not understanding).
- Serving content only via JavaScript: crawlers then see empty pages.
- Maintaining the bot list once and never revisiting it, although new crawlers appear all the time.
What to do, in three steps
- Check how AI systems see your site today, with the AI check.
- Provide content: create an llms.txt with the generator.
- Govern access deliberately: allow answer crawlers, optionally limit training crawlers.
How this page stays current
The list of AI crawlers grows almost every week. That is why the section above is refreshed automatically from a maintained source, and this page’s modified date moves with it.
Ready to get found by AI?
Check in seconds how well AI systems read your website or list yourself in the directory for free.
List your website for free →