Can AI crawlers see your site?
Check in seconds whether GPTBot, ClaudeBot, PerplexityBot and search crawlers can actually reach your pages — robots.txt rules plus a live server-response probe. Free, no account.
- Live probes, not just robots.txt
- No account required
- Checked URLs are not stored
- Free to use
Choose your problem
Why it matters
AI assistants and AI search now decide whether your site gets cited. Blocks hide in two layers: your robots.txt rules, and server/CDN-level bot protection (like Cloudflare's default AI-crawler blocking) that robots.txt checkers never see. Our checker tests both and tells you exactly which bot is blocked where. For each crawler's verified user-agent strings and official documentation, see the crawler directory. Browse the ai crawler directory.
Guides
- AI search visibility (GEO): making sure the assistants can actually read you
GEO / AI-search visibility has a prerequisite people skip: can the AI engines fetch your pages at all? The reachability checklist and how to verify it.
- Why your robots.txt says 'allowed' but AI crawlers still can't reach you
robots.txt allows AI crawlers but they still can't reach you? The block is at your CDN/server layer. How to confirm which gatekeeper is saying no.
- How to opt out of AI training without hurting your Google Search ranking
Google-Extended lets you opt out of Gemini training with zero effect on Google Search ranking (per Google). How to set it — and why not to block Googlebot by mistake.
- GPTBot vs OAI-SearchBot vs ChatGPT-User: which one to allow
OpenAI's GPTBot (training), OAI-SearchBot (search), and ChatGPT-User (user fetch) are three separate crawlers. Which to allow to stay findable without being trained on.
- Is your site blocked from ChatGPT, Claude, and Perplexity? How to check both layers
Two layers decide whether ChatGPT/Claude/Perplexity can reach your site — robots.txt and your CDN. Most checkers see only one. How to check both.
Requirement news
- Cloudflare's Sept 15 default AI-crawler blocking: what site owners should check
Cloudflare's Sept 15 2026 default blocking of AI crawlers on ad-supported sites means robots.txt can say 'allowed' while the edge says 'blocked' — here's how to check both.