Skip to content

robots.txt for AI crawlers: which bots to allow

Updated · 6 min read

Short answer

Allow the search crawlers that fetch pages for AI answers (OAI-SearchBot, PerplexityBot, Googlebot, Bingbot, Claude-SearchBot) and the user fetchers (ChatGPT-User, Perplexity-User), or the assistants cannot read or cite you. Training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot) are a separate choice: blocking them does not remove you from search-based answers, but future models will know less about you.

Three kinds of AI bot

Search crawlers index pages so an assistant can search them while answering. User fetchers open a specific page because a person asked the assistant about it. Training crawlers collect text that future models learn from. Each kind has its own user-agent token, so robots.txt can treat them differently.

The user-agents that matter (as of November 2026)

  • OAI-SearchBot (OpenAI, search): pages shown and cited in ChatGPT search.
  • ChatGPT-User (OpenAI, user fetcher): pages ChatGPT opens for a user.
  • GPTBot (OpenAI, training): text for future OpenAI models.
  • PerplexityBot (Perplexity, search): pages Perplexity cites in answers.
  • Perplexity-User (Perplexity, user fetcher): pages opened for a user’s question.
  • Googlebot (Google, search): Google Search, and therefore AI Overviews and AI Mode.
  • Google-Extended (Google, training token): whether your content may be used for Gemini models and grounding. It is a robots.txt token, not a separate crawler.
  • Bingbot (Microsoft, search): Bing and Copilot; ChatGPT search also draws on Bing’s index.
  • ClaudeBot (Anthropic, training) and Claude-SearchBot (Anthropic, search).
  • Applebot-Extended (Apple, training token) and CCBot (Common Crawl, open datasets many models train on).

Check and fix your robots.txt

  1. Open yoursite.com/robots.txt

    Read every User-agent group. A bot follows the group that names it; only if none does, it follows the User-agent: * group.

  2. Look for a blanket block

    A "Disallow: /" under User-agent: * blocks every bot that is not named elsewhere, including the AI search crawlers.

  3. Allow the search crawlers and user fetchers by name

    Add a group for OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and Claude-SearchBot with "Allow: /", and keep Googlebot and Bingbot allowed.

  4. Decide on training separately

    If you prefer not to be used for training, disallow GPTBot, Google-Extended, ClaudeBot, Applebot-Extended and CCBot. That is a legitimate choice; it does not block search-based answers.

  5. Check the firewall too

    Some CDNs and bot-protection settings block AI crawlers before robots.txt is even read. Make sure verified AI search bots are not challenged.

  6. Test it

    Run the free AI-readiness check at seenby.danixsoft.com/ai-readiness: it reads your robots.txt for 12 AI crawlers and says which are blocked.

An example robots.txt

This allows AI search and user fetchers, opts out of model training, and lists the sitemap. Remove the training block if you want future models to learn about you.

# AI search and user fetchers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
Allow: /

# Model training: opted out (optional)
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Questions

If I block GPTBot, will ChatGPT stop mentioning me?
Not from search. ChatGPT search uses OAI-SearchBot and Bing’s index, so blocking only GPTBot keeps you available for search-based answers. It does mean future OpenAI models learn less about you from your own site.
Do AI crawlers obey robots.txt?
The major providers say their named crawlers do. User fetchers acting for a person are treated differently by some providers, and unknown scrapers may ignore robots.txt entirely; use your firewall for those.
Does Google-Extended affect AI Overviews?
No. AI Overviews are part of Google Search and use Googlebot. Google-Extended controls use for Gemini models and grounding outside Search.