robots.txt for AI crawlers: which bots to allow
Updated · 6 min read
Short answer
Allow the search crawlers that fetch pages for AI answers (OAI-SearchBot, PerplexityBot, Googlebot, Bingbot, Claude-SearchBot) and the user fetchers (ChatGPT-User, Perplexity-User), or the assistants cannot read or cite you. Training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot) are a separate choice: blocking them does not remove you from search-based answers, but future models will know less about you.
Three kinds of AI bot
Search crawlers index pages so an assistant can search them while answering. User fetchers open a specific page because a person asked the assistant about it. Training crawlers collect text that future models learn from. Each kind has its own user-agent token, so robots.txt can treat them differently.
The user-agents that matter (as of November 2026)
- OAI-SearchBot (OpenAI, search): pages shown and cited in ChatGPT search.
- ChatGPT-User (OpenAI, user fetcher): pages ChatGPT opens for a user.
- GPTBot (OpenAI, training): text for future OpenAI models.
- PerplexityBot (Perplexity, search): pages Perplexity cites in answers.
- Perplexity-User (Perplexity, user fetcher): pages opened for a user’s question.
- Googlebot (Google, search): Google Search, and therefore AI Overviews and AI Mode.
- Google-Extended (Google, training token): whether your content may be used for Gemini models and grounding. It is a robots.txt token, not a separate crawler.
- Bingbot (Microsoft, search): Bing and Copilot; ChatGPT search also draws on Bing’s index.
- ClaudeBot (Anthropic, training) and Claude-SearchBot (Anthropic, search).
- Applebot-Extended (Apple, training token) and CCBot (Common Crawl, open datasets many models train on).
Check and fix your robots.txt
Open yoursite.com/robots.txt
Read every User-agent group. A bot follows the group that names it; only if none does, it follows the User-agent: * group.
Look for a blanket block
A "Disallow: /" under User-agent: * blocks every bot that is not named elsewhere, including the AI search crawlers.
Allow the search crawlers and user fetchers by name
Add a group for OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and Claude-SearchBot with "Allow: /", and keep Googlebot and Bingbot allowed.
Decide on training separately
If you prefer not to be used for training, disallow GPTBot, Google-Extended, ClaudeBot, Applebot-Extended and CCBot. That is a legitimate choice; it does not block search-based answers.
Check the firewall too
Some CDNs and bot-protection settings block AI crawlers before robots.txt is even read. Make sure verified AI search bots are not challenged.
Test it
Run the free AI-readiness check at seenby.danixsoft.com/ai-readiness: it reads your robots.txt for 12 AI crawlers and says which are blocked.
An example robots.txt
This allows AI search and user fetchers, opts out of model training, and lists the sitemap. Remove the training block if you want future models to learn about you.
# AI search and user fetchers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
Allow: /
# Model training: opted out (optional)
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlQuestions
- If I block GPTBot, will ChatGPT stop mentioning me?
- Not from search. ChatGPT search uses OAI-SearchBot and Bing’s index, so blocking only GPTBot keeps you available for search-based answers. It does mean future OpenAI models learn less about you from your own site.
- Do AI crawlers obey robots.txt?
- The major providers say their named crawlers do. User fetchers acting for a person are treated differently by some providers, and unknown scrapers may ignore robots.txt entirely; use your firewall for those.
- Does Google-Extended affect AI Overviews?
- No. AI Overviews are part of Google Search and use Googlebot. Google-Extended controls use for Gemini models and grounding outside Search.