Does your robots.txt block the AI crawlers?
Many sites lock out GPTBot, ClaudeBot & PerplexityBot without knowing it and show up less in AI answers as a result. We check your robots.txt against 20 AI crawlers using the rules Google applies (incl. the inheritance trap) and whether your llms.txt is reachable.
Reads only your public robots.txt & llms.txt · result cached for 6 hours · free
Why this matters
- AI search & live fetch: if you block OAI-SearchBot, PerplexityBot or Claude-User, AI cannot cite your content. Other providers get named instead.
- The inheritance trap: a bot with its own
User-agentgroup ignores all*rules entirely. This is a common and easily overlooked misconfiguration. - Training is your call: blocking training crawlers (GPTBot, CCBot …) is legitimate, but blocking search and live bots costs you direct visibility.
Stay AI-visible
Get the most important GEO & llms.txt strategies and product updates straight to your inbox. No spam, unsubscribe anytime with one click.