Should you block GPTBot and ClaudeBot?
Last updated:
Short answer: for most businesses, no - and it matters enormously which bot you block. The crawlers that train AI models are different from the ones that decide whether you appear in ChatGPT, Claude and Perplexity answers. Block the wrong one and you quietly remove yourself from the results you were hoping to win.
The distinction that matters
Every major AI company now runs separate crawlers for separate jobs, and gives them separate names precisely so you can treat them differently:
- Training crawlers collect content to teach the model. Blocking them protects your content from being learned, and costs you nothing in visibility.
- Search crawlers build the index the assistant consults when answering. Blocking them removes you from those answers.
- User-initiated fetchers visit a page because a person asked a question about it right now. Blocking them means the assistant cannot read your page even when a user explicitly asks.
A publisher worried about their archive being used for training and a shop that wants to be recommended want opposite things - and both are achievable, because the bots are separable.
The main AI crawlers, and what blocking each one costs you
Each row below reflects the operator's own published documentation. Bots change; the date at the top of this page is when we last checked. This covers the engines most people ask about - it is not an exhaustive list of every AI-related crawler on the web. Others you may see in your logs include Meta-WebIndexer, Meta-ExternalFetcher, Applebot-Extended, Bytespider, Amazonbot, cohere-ai and MoonshotBot; check the operator's own documentation before deciding on any of them.
| User-agent | Run by | Type | What it does | If you block it |
|---|---|---|---|---|
GPTBot | OpenAI | Training | Crawls content to train OpenAI's foundation models. | Your content is excluded from future model training. No effect on ChatGPT search. |
OAI-SearchBot | OpenAI | Search | Surfaces websites in ChatGPT's search features. | OpenAI states opted-out sites “will not be shown in ChatGPT search answers, though can still appear as navigational links”. This is the one that costs you visibility. |
ChatGPT-User | OpenAI | User-initiated | Fetches a page when a ChatGPT user or a Custom GPT asks for it. | Not automated crawling. OpenAI notes robots.txt rules may not apply to it, as it acts on a person's request. |
OAI-AdsBot | OpenAI | Ads | Checks the safety of pages submitted as advertisements on ChatGPT. | No effect on training or search. OpenAI states data it collects is not used to train models. |
ClaudeBot | Anthropic | Training | Collects web content to improve Anthropic's generative AI models. | Signals that your future material should be excluded from model training datasets. |
Claude-SearchBot | Anthropic | Search | Navigates the web to improve search result quality for Claude users. | Prevents Anthropic indexing your content for search. Costs you Claude visibility. |
Claude-User | Anthropic | User-initiated | Retrieves a page when a Claude user asks a question that needs web access. | Claude cannot retrieve your content in response to a user's query - even when someone explicitly asks about you. |
PerplexityBot | Perplexity | Search | Surfaces and links websites in Perplexity results. Perplexity states it is not used to crawl content for AI foundation models. | Your site stops appearing in Perplexity results. |
Perplexity-User | Perplexity | User-initiated | Visits a page to answer a specific user question, and links it in the response. | Little or nothing. Perplexity states this fetcher “generally ignores robots.txt rules”, because a person asked for the page. |
Google-Extended | Training + grounding | Controls whether your content trains Gemini models AND whether it can ground Gemini answers at prompt time. Not a crawler in its own right - it is a control token. | No effect on Google Search - Google states it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal”. But it DOES switch off grounding, so you lose visibility in Gemini's answers. | |
meta-externalagent | Meta | Training | Crawls to train Meta's foundation AI models, and to index content directly. | Excludes your content from Meta's AI training and indexing. |
CCBot | Common Crawl | Training | Crawls the open web for the Common Crawl public dataset, which many AI companies use as a training source. | Keeps your content out of a dataset several model builders draw on. |
What we recommend for most businesses
If you sell something, provide a service, or want to be found: allow everything. You are trying to be recommended, and the cost of being invisible in AI answers is far larger than the theoretical cost of your public marketing pages contributing to a model.
# Allow every AI crawler (recommended if you want to be found) User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
A permissive User-agent: * already allows them all - naming each bot is optional and mostly a statement of intent. What genuinely matters is not accidentally blocking them.
If you want to opt out of training but stay visible
This is the sensible middle ground for publishers, and the reason the bots have separate names. Block the training crawlers, keep the search and user-initiated ones:
# Opt out of model training, stay visible in AI answers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # Google-Extended is a judgement call, not a free win - see the note below. # User-agent: Google-Extended # Disallow: / # Keep these allowed - they are what put you in the answers User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: /
Google-Extended is the one real trade-off here, which is why it is commented out above. Google states it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search” - so your blue-link rankings are genuinely safe. But the same control also governs grounding, which is how Gemini pulls live content into an answer. Block it and you keep your Google rankings while removing yourself from Gemini's answers. Decide which you care about more.
Perplexity-User is listed above for completeness, but Perplexity states it generally ignores robots.txt because a person requested the page - so allowing it is a statement of intent rather than a switch.
Three mistakes worth avoiding
- Blocking every bot with an AI-sounding name. The most common version of this is disallowing
OAI-SearchBotalongsideGPTBot, which removes you from ChatGPT search results while doing nothing extra for your content. - Copying a robots.txt from a publisher. News organisations block aggressively because their archive is the product. If your website is marketing for a business, their reasoning does not transfer.
- Assuming robots.txt is enforcement. It is a request that well-behaved crawlers honour. It is not access control, and it will not stop anyone determined.
Blocking is not the only lever
Letting crawlers in is necessary but not sufficient. If your content only appears after JavaScript runs, or your headings are decorative rather than structural, a crawler that is allowed in still leaves with very little. That is what our AI Search Readiness checks look at - see the full list, or run a free checkup on your own site.
For the related question of describing your site to AI systems in plain text, see our llms.txt guide.