Control AI crawlers: llms.txt, robots.txt & the current bot list

A growing share of visits to your website come not from humans but from AI crawlers. Understanding them lets you decide deliberately who may read your content — and ensures the right systems get the right information. This guide separates myth from practice and shows the currently known bots.

Three kinds of AI crawlers

Not every AI bot has the same purpose. To decide wisely, distinguish three types:

1. Training/dataset crawlers collect content to train future models (e.g. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended). Blocking protects content but lowers the chance of being "learned".
2. Search/index crawlers for answer engines build the index that AI answers cite from (e.g. OAI-SearchBot, PerplexityBot). Blocking these directly costs AI visibility.
3. On-demand "user" fetches retrieve a page only when a user asks the AI about it (e.g. ChatGPT-User, Perplexity-User). This is essentially real, intent-rich traffic.

robots.txt vs. llms.txt — the key difference

They work together but solve different jobs:

• robots.txt controls access — it tells a crawler which paths it may fetch. It is not content.
• llms.txt provides content — it offers your verified core knowledge in machine-readable form so AI systems understand and cite you correctly. → What is GEO?

In short: robots.txt is the door, llms.txt is the well-organized shop window behind it.

Block or allow? A clear trade-off

Most businesses want to be found and recommended. Then: allow answer/search crawlers and on-demand fetches and additionally provide an llms.txt. You can selectively block training crawlers to protect content — but be aware this also lowers your brand's long-term "familiarity" inside models. Blanket-blocking everything is the most common costly mistake.

How to control access (robots.txt)

# Allow answer crawlers (recommended for visibility)
User-agent: *
Allow: /

# Example: exclude one specific training crawler
User-agent: GPTBot
Disallow: /

# Also point to your content source
Sitemap: https://your-domain.com/sitemap.xml

Note: a rule only applies to the exact named user-agent. New bots appear constantly — so a maintained reference list is worth checking.

Currently known AI crawlers (live)

So this overview does not go stale, we pull it live from a continuously maintained reference — it updates here automatically:

Currently 164 AI crawler user-agents are documented. This list refreshes automatically as new bots appear:

AddSearchBot  ·  AgentTimes  ·  AI2Bot  ·  AI2Bot-DeepResearchEval  ·  Ai2Bot-Dolma  ·  aiHitBot  ·  AIWebIndex  ·  amazon-kendra  ·  amazon-QBusiness  ·  Amazonbot  ·  AmazonBuyForMe  ·  Amzn-SearchBot  ·  Amzn-User  ·  Andibot  ·  Anomura  ·  anthropic-ai  ·  ApifyBot  ·  ApifyWebsiteContentCrawler  ·  Applebot  ·  Applebot-Extended  ·  Aranet-SearchBot  ·  atlassian-bot  ·  Awario  ·  AzureAI-SearchBot  ·  bedrockbot  ·  bigsur.ai  ·  Bravebot  ·  Brightbot  ·  Brightbot 1.0  ·  BuddyBot  ·  Bytespider  ·  CCBot  ·  Channel3Bot  ·  ChatGLM-Spider  ·  ChatGPT Agent  ·  ChatGPT-User  ·  Claude-Code  ·  Claude-SearchBot  ·  Claude-User  ·  Claude-Web  ·  ClaudeBot  ·  Cloudflare-AutoRAG  ·  CloudVertexBot  ·  Code  ·  cohere-ai  ·  cohere-training-data-crawler  ·  Cotoyogi  ·  CragCrawler  ·  Crawl4AI  ·  Crawlspace  ·  Cursor  ·  Datenbank Crawler  ·  DeepSeekBot  ·  Devin  ·  Diffbot  ·  DuckAssistBot  ·  Echobot Bot  ·  EchoboxBot  ·  ExaBot  ·  FacebookBot  ·  facebookexternalhit  ·  Factset_spyderbot  ·  FirecrawlAgent  ·  FriendlyCrawler  ·  GeistHaus-PageFetcher  ·  Gemini-Deep-Research  ·  Google-Agent  ·  Google-CloudVertexBot  ·  Google-Extended  ·  Google-Firebase  ·  Google-Gemini-CLI  ·  Google-NotebookLM  ·  GoogleAgent-Mariner  ·  GoogleAgent-URLContext  ·  GoogleOther  ·  GoogleOther-Image  ·  GoogleOther-Video  ·  GPTBot  ·  HenkBot  ·  iAskBot  ·  iaskspider  ·  iaskspider/2.0  ·  IbouBot  ·  ICC-Crawler  ·  ImagesiftBot  ·  imageSpider  ·  img2dataset  ·  ISSCyberRiskCrawler  ·  kagi-fetcher  ·  Kangaroo Bot  ·  Kimi-User  ·  KlaviyoAIBot  ·  KunatoCrawler  ·  laion-huggingface-processor  ·  LAIONDownloader  ·  LCC  ·  LinerBot  ·  Linguee Bot  ·  LinkupBot  ·  Manus-User  ·  meta-externalagent  ·  Meta-ExternalAgent  ·  meta-externalfetcher  ·  Meta-ExternalFetcher  ·  meta-webindexer  ·  MistralAI-User  ·  MistralAI-User/1.0  ·  Mozilla-Tabstack  ·  MyCentralAIScraperBot  ·  NagetBot  ·  netEstate Imprint Crawler  ·  newsai  ·  NotebookLM  ·  NovaAct  ·  OAI-SearchBot  ·  omgili  ·  omgilibot  ·  OpenAI  ·  opencode  ·  Operator  ·  PanguBot  ·  Panscient  ·  panscient.com  ·  Perplexity-User  ·  PerplexityBot  ·  PetalBot  ·  PhindBot  ·  Poggio-Citations  ·  Poseidon Research Crawler  ·  QualifiedBot  ·  Querit-SearchBot  ·  QueritBot  ·  QuillBot  ·  quillbot.com  ·  SBIntuitionsBot  ·  Scrapy  ·  SemrushBot-OCOB  ·  SemrushBot-SWA  ·  Shap-User  ·  ShapBot  ·  Sidetrade indexer bot  ·  Spider  ·  TavilyBot  ·  Terra Cotta  ·  TerraCotta  ·  Thinkbot  ·  TikTokSpider  ·  Timpibot  ·  TongyiBot  ·  Trae  ·  TwinAgent  ·  UseAI  ·  VelenPublicWebCrawler  ·  WARDBot  ·  Webzio-Extended  ·  webzio-extended  ·  wpbot  ·  WRTNBot  ·  YaK  ·  YandexAdditional  ·  YandexAdditionalBot  ·  YiyanBot  ·  YouBot  ·  ZanistaBot

Source: the continuously maintained ai.robots.txt project — refreshed automatically · last change detected 10.07.2026.

Common mistakes

• Blocking all bots out of caution — then wondering about missing AI visibility.
• Mistaking robots.txt for a content solution (it governs access, not understanding).
• Serving content only via JavaScript — crawlers then see empty pages.
• Maintaining the bot list once and never revisiting it, although new crawlers appear all the time.

Set up right in 3 steps

1. Check how AI systems see your site today — with the AI check.
2. Provide content: create an llms.txt with the generator.
3. Govern access deliberately: allow answer crawlers, optionally limit training crawlers.

Why this page stays current

The list of AI crawlers grows almost weekly. That is why the section above is refreshed automatically from a maintained source; when the list changes, this page's modified date moves with it. So you stay current, and search engines reliably detect fresh content.