Does your robots.txt block the AI crawlers?
Many sites lock out GPTBot, ClaudeBot & PerplexityBot without knowing it — and never appear in AI answers. We check your robots.txt against 20 AI crawlers with real Google semantics (incl. the notorious inheritance trap) and whether your llms.txt is reachable.
Reads only your public robots.txt & llms.txt · result cached for 6 hours · free
Why this matters
- AI search & live fetch: if you block OAI-SearchBot, PerplexityBot or Claude-User, AI cannot cite your content — your competitors get recommended instead.
- The inheritance trap: a bot with its own
User-agentgroup ignores all*rules entirely — the most common hidden misconfiguration. - Training is your call: blocking training crawlers (GPTBot, CCBot …) is legitimate — but blocking search and live bots costs you direct visibility.
Stay AI-visible
Get the most important GEO & llms.txt strategies and product updates straight to your inbox. No spam, unsubscribe anytime with one click.