robots.txt Explained: What Belongs in It and What Does Not

What the file does

Search engines and other crawlers read the robots.txt before they go through a website. It states which paths they should avoid and where the sitemap is.

A simple example

User-agent: *
Disallow: /intern/
Sitemap: https://example.de/sitemap.xml

Important notes

• Not a security measure: Anyone who does not follow it can still open the page. Confidential content belongs behind a login.
• Blocking does not mean removal from the index: A blocked page can still appear in the results with its address. To exclude a page, use noindex.
• Blocking by accident: A “Disallow: /” blocks the whole website. This happens after relaunches more often than you would think.

Structure in detail

The file consists of blocks. A block begins with a User-agent line naming the crawler it applies to, and below it contains Disallow or Allow lines. The asterisk applies to all crawlers. A Disallow: with no path means that nothing is blocked. Paths are relative to the domain and are case-sensitive. The file must sit in the root directory, that is at yourdomain.com/robots.txt; in subfolders it is ignored. Each subdomain needs its own file.

What you can block and what you cannot

• Sensible: internal search results pages, basket and checkout, filter combinations with parameters, test areas that should not be publicly findable anyway.
• Do not block: stylesheets and scripts needed for rendering. Google renders pages, and blocked resources can make them harder to understand.
• Do not rely on robots.txt: for confidential data. The file is publicly readable and so even reveals the paths you wanted to hide.

Blocking or noindex: making the decision

This misunderstanding is common: if you block a page via robots.txt, you stop Google from visiting it, not from listing it. If Google knows the address through external links, it can appear in the index without a description. If a page should not appear, you need a noindex in the head section or in the server's response, and the page must not be blocked for this, otherwise Google never sees the noindex. If you want to save crawl effort, for instance with thousands of filter pages, blocking is the right tool.

The mistake after a relaunch

During development, agencies often block the test version completely. If the file is copied across when the site goes live, the new website stays invisible to search engines. After every relaunch, check the robots.txt and the noindex setting in the content management system first. One indication is pages suddenly missing from the Search Console report.

Checking steps

1. Open yourdomain.com/robots.txt in your browser. If an error page appears, no file exists, which is permissible.
2. Look for “Disallow: /” with no further path.
3. Check that the sitemap line is correct.
4. Test important addresses with the URL Inspection in Search Console.

Test changes carefully

Because a single wrong entry can exclude large parts of the website, never change the file without checking. Save a copy of the old version beforehand, then test typical addresses in the Search Console URL Inspection and monitor the indexing report over the following days. A drop in indexed pages shortly after a change is a clear warning sign that you should follow up straight away.

AI crawlers

AI systems also use their own crawlers, such as GPTBot, ClaudeBot or PerplexityBot. If you want to be cited in AI answers, you should not block them without intending to. More in AI crawlers in robots.txt.