robots.txt Generator
Generate an AI-crawler-aware robots.txt: allow GPTBot, ClaudeBot, PerplexityBot and friends while keeping control over sensitive paths.
Frequently Asked Questions
Which AI crawlers are included?
GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), CCBot (Common Crawl), Bytespider (ByteDance), DeepSeekBot, Meta-ExternalAgent and more.
What's the difference between blocking training vs search AI?
Training bots (GPTBot, ClaudeBot, Google-Extended) scrape content to train models. Search AI bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetch pages to cite them in live answers. Blocking search AI removes you from AI answers entirely.
Will allowing AI crawlers hurt my SEO?
No. AI crawlers are separate from Googlebot. Allowing them does not affect your Google ranking — it only decides whether ChatGPT, Perplexity and friends can read and cite you.
Why is there a User-agent line for each bot?
robots.txt matches rules per user-agent. Listing each bot explicitly is the only reliable way to control access per engine; a single wildcard rule may be ignored by some crawlers.
Where do I put this file?
At your domain root, exactly at /robots.txt. Most hosts let you upload it directly; verify with our AI Crawler Checker afterward.