How Search Engines Work
Search engines such as Google and Bing work in three main stages: they discover pages on the web (crawling), store and understand them (indexing), and then decide which of them best answers each search (ranking). Everything you do in SEO is aimed at making one of those three stages work better for your website.
Stage 1: Discovering pages
Google uses automated programs called crawlers, or Googlebot, to fetch web pages. Crawlers find new pages by following links from pages they already know, by reading XML sitemaps and by processing URLs submitted in Google Search Console. If no link points to a page and it is not in a sitemap, Google may never find it. That is why Internal Linking and a clear Site Architecture matter so much.
You can read more about this first stage in the entry on Crawling.
Stage 2: Understanding and storing pages
Once a page is fetched, Google processes its HTML, runs any JavaScript (Rendering) and works out what the page is about. It looks at the text, headings, images, links, structured data and much more. If the page is useful and allowed to be indexed, it is stored in the Google index, a huge database of billions of pages.
Not every crawled page gets indexed. Pages can be left out because they are duplicates, low quality, blocked by a Noindex Tag or simply not considered valuable enough. The Indexing entry explains how to check and fix this, and my guide to the Discovered currently not indexed fix covers one of the most common problems.
Stage 3: Ranking and serving results
When someone searches, Google looks through its index and orders the matching pages using hundreds of signals, known as Google Ranking Factors. The most important ones relate to:
- Relevance: does the page match the words and the Search Intent behind the query?
- Quality: is the content helpful, accurate and written by someone with real experience?
- Authority: do other trusted websites link to and mention it?
- Usability: does the page load quickly and work well on a phone?
- Context: the searcher’s location, language and device.
The results are then shown on the SERP (Search Engine Results Page), which today can include ads, maps, videos, Featured Snippets and AI Overviews as well as the classic blue links.
How AI is changing search
Search engines now use machine learning at every stage, from understanding the meaning of a query to writing AI-generated summaries. AI assistants such as ChatGPT also search the web and cite sources. The fundamentals have not changed, though: content still has to be found, understood and trusted before it can appear. The AI SEO entry explains what to do differently.
What this means for your website
To perform well in search, make sure each stage works in your favour:
- Make pages discoverable with clean internal links and an up-to-date sitemap.
- Make pages indexable by avoiding accidental noindex tags, robots.txt blocks and duplicate URLs.
- Make pages worth ranking with content that answers the search better than competing pages.
- Build trust through links, reviews and mentions from other websites.
If you are just starting out, read what SEO is and how to get your website on Google next.
To see which signals decide the order of results, read the complete guide to Google ranking factors.
Further reading: Google’s in-depth guide to how Search works.
Key Takeaways
- Search engines work in three stages: crawling, indexing and ranking.
- Pages must be discoverable through links or sitemaps before they can be indexed.
- Not every crawled page is indexed; quality and uniqueness matter.
- Rankings balance relevance, quality, authority, usability and context.
- AI now runs through every stage, but the fundamentals still apply.
Related Terms
Need a second opinion on your website? Ask for a free website SEO check, or keep learning in the SEO Encyclopedia.