AI Crawler Checker
Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot…) can access your site, parse your robots.txt rules, and see your Common Crawl archive status.
Frequently Asked Questions
What is an AI crawler?
AI crawlers are bots from AI companies (OpenAI, Anthropic, Google…) that visit your site to feed their models and AI search features. GPTBot feeds ChatGPT training, OAI-SearchBot feeds ChatGPT's web search.
Should I block AI crawlers?
It depends on your goal. Block them to protect original content from AI training. Allow them if you want your site cited in AI answers — AI search engines are becoming a real traffic source.
Why does my site return 200 to GPTBot but I'm not in ChatGPT?
Returning HTTP 200 means the crawler can technically fetch your page. Being cited in AI answers additionally requires the AI to judge your content relevant — that's a different, slower process.
How does the checker test crawlers?
We fetch your site's robots.txt to read the authoritative rules, then probe your homepage with each crawler's real User-Agent string. Common Crawl status is queried from the public index API.
What if a site blocks by IP, not User-Agent?
Some WAFs (like Cloudflare Bot Management) block datacenter IPs regardless of User-Agent. Our probe runs from Cloudflare's edge, so such blocks may show as timeout — we mark results as advisory and rely on robots.txt as the authoritative signal.