Crawling, indexing, ranking: how Google works

The three steps in one sentence

A search engine works in three steps: it finds and fetches pages (crawling), it analyses the content and stores it in its database (indexing), and when someone searches, it sorts the matching pages by relevance and quality (ranking). Each step can fail, and each has different causes. So if you want to solve a visibility problem, you first need to know where the page gets stuck.

Step 1: Crawling – being found and fetched

Google runs programs that fetch web pages, known as Googlebot. It follows links from known pages to new ones and also reads your sitemap, if you have provided one. Crawling is fetching, not yet evaluating.

Typical reasons why a page is not fetched:

• No other page links to it, and it is not in any sitemap.
• The robots.txt file blocks the area. Details in robots.txt explained.
• The server does not respond, or responds very slowly.
• The page sits behind a login.

A sitemap helps with finding pages. How to create and submit one is covered in Creating and submitting an XML sitemap. Good internal linking is another way of guiding the crawler, see Internal linking.

Step 2: Indexing – being understood and stored

After fetching a page, Google analyses what it is about: text, headings, images, structured data, language, links. It then decides whether to add the page to its index. The index is the database from which search results are put together. The important point: fetched does not automatically mean stored. Google does not include every page.

Common reasons for “crawled, but not indexed”:

• The page carries a noindex signal, often carried over by mistake from the test environment after a relaunch.
• The content is very thin or almost identical to another page.
• A canonical tag points to a different address, see Canonical tag explained.
• The page is only built by scripts and is hard for the crawler to read.

The free Search Console shows how Google sees your pages. Setting it up is described in Setting up Google Search Console.

Step 3: Ranking – being sorted

A page can only appear in a search once it is in the index. Google then assesses how well it matches the query. Which signals play a part is only partly public; we have summarised what is established and what is not in Google ranking factors: what we know. What is established is that Google takes the purpose of the search into account (whether someone wants information, wants to compare or wants to buy); more on this in Search intent explained. Google does not publish how it weights the individual signals.

Rankings are also not the same for everyone. Location, device and previous search behaviour can change the results. A statement such as “position 3” is therefore always a snapshot.

Where is your page stuck? A quick diagnosis

The “site:” command is only a rough guide. The “Pages” report in Search Console is more reliable, because it also gives the reason why pages are not indexed.

What you can do straight away

1. Create a Search Console account and verify your domain.
2. Check your five most important pages with the URL inspection tool: are they indexed?
3. Submit the sitemap and check that the number of pages it has detected is right.
4. Link every important page from at least one other page on your website.
5. Check whether a noindex tag or a robots.txt block is still in place somewhere it does not belong.

As for the order: make sure of crawling and indexing first, then work on ranking. Improving content that is not in the index at all achieves nothing. If you get stuck after these checks, the SEO and GEO page gives an overview of how we approach such questions, or you can write to us via Contact.