AI crawlers in robots.txt: GPTBot, ClaudeBot and others

The most important identifiers

• GPTBot (OpenAI): collects content, among other things for training models.
• OAI-SearchBot (OpenAI): for search in ChatGPT.
• ChatGPT-User (OpenAI): fetches pages when users trigger it in the chat.
• ClaudeBot (Anthropic) as well as Claude-User and Claude-SearchBot.
• PerplexityBot (Perplexity): for AI search.
• Google-Extended: controls use for Google’s AI products, but has no influence on normal Google Search.
• Applebot-Extended (Apple) and CCBot (Common Crawl, the data source for many models).

Allow or block?

This is a decision between visibility and control. If you want to appear in AI answers, you should at least allow the search and answer services. If you do not want your content used for training, block the relevant crawlers specifically. How the services are divided up changes, so check the providers’ information.

Example of allowing

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Separating training and search: a worked example

Suppose a trade business wants to appear in AI answers but does not want to make its texts available for training models. Its robots.txt then needs two groups of entries: a block for the crawlers that mainly collect training data, and an allowance for the crawlers of the search functions. The file would be built like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

The assignment of identifiers to their purpose comes from the providers and can change. Before any adjustment, therefore, check the current documentation from OpenAI, Anthropic, Perplexity and Google rather than relying on an old list.

Typical mistakes

• A blanket block for all bots: User-agent: * with Disallow: / also shuts out search engines. This often happens when a test version of the website was not cleaned up after launch.
• Confusing Google-Extended with Googlebot: Google-Extended only concerns use for Google’s AI products. If you block Googlebot instead, you disappear from normal search.
• Typos in the identifier: the identifier must match the name the provider gives. With a typo, the entry has no effect and the crawler keeps accessing your site.
• Relying on robots.txt alone as a barrier: the file is an agreement among polite participants. If you really need to protect content, you need access protection on the server.

Checks after every change

1. Open your-domain.com/robots.txt in the browser and check that the file can be reached and shows the expected content.
2. Read it group by group: which identifier stands above which rule? A rule always applies to the identifier directly above it.
3. Look in the server logs to see whether the crawlers still come after the change or stay away. This shows whether the permission actually arrives.
4. Note the date of the change. If mentions in AI answers change later, you can at least place the connection.

A decision aid

For businesses that win customers through enquiries, the benefit of visibility usually outweighs the rest. For publishers, picture agencies or providers of paid content, the calculation can look different: there the main question is whether content may be exploited without anything in return. Both positions are legitimate; what matters is that the choice is deliberate and clearly reflected in the file. How legally binding such a reservation is, for example under copyright law, is a question for legal advice.

The difference between retrieval and training

If an assistant fetches a page because a user has just asked about it, that is something different from a crawler collecting large amounts of text in the background for a later model. The identifiers make exactly this distinction possible. If you block only the training crawlers, you can still appear in answers with a source reference. Whether that is implemented this way in an individual case depends on the provider, as some services use the same identifier for several purposes. So read the providers’ notes on every identifier you enter.

Important

An entry in robots.txt is a request, not a block. Reputable crawlers abide by it. Basics under robots.txt explained.