Indexing

Indexing is the process of a search engine storing a page in its database, called the index, so it can appear in search results. After Google crawls a page, it analyses the content and decides whether to add it to the index. Only indexed pages can rank.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

How indexing works

Once Googlebot has fetched and, if needed, rendered a page, Google’s systems work out what it is about, which language it is in, whether it duplicates another page and how useful it is. If Google finds several similar pages, it picks one as the canonical version and indexes that one (see Canonical Tag). The rest are treated as duplicates.

Google does not promise to index every page it crawls. With billions of pages online, it is selective, especially with new sites, low-quality pages and near-duplicates.

How to check if a page is indexed

  • Use the URL Inspection tool in Google Search Console. It tells you whether the URL is on Google and, if not, why.
  • Review the Pages report for a site-wide view of indexed and excluded URLs.
  • A quick site: search (for example, site:yourdomain.co.uk/page) gives a rough idea, but it is not reliable for detail.

Common reasons pages are not indexed

  • Excluded by noindex: a Noindex Tag tells Google not to index the page.
  • Blocked by robots.txt: Google cannot crawl it, so it cannot properly index it.
  • Duplicate without user-selected canonical: Google chose a different URL as the main version (Duplicate Content).
  • Alternate page with proper canonical tag: working as intended.
  • Page with redirect or Not found (404): the URL no longer holds content.
  • Crawled, currently not indexed: Google looked at the page and decided not to index it yet, often because of quality or duplication.
  • Discovered, currently not indexed: Google knows the URL but has not crawled it yet, often because of crawl priorities or site quality.

The last two are the most common frustrations. My guide to the Discovered currently not indexed fix explains how to tackle them.

How to get pages indexed

  1. Make sure the page is not blocked by robots.txt or a noindex tag.
  2. Link to it from relevant, already-indexed pages (Internal Linking).
  3. Include it in your XML Sitemap.
  4. Make the content unique, substantial and genuinely helpful; avoid Thin Content.
  5. Use a self-referencing canonical tag.
  6. Request indexing with the URL Inspection tool for important new or updated pages.
  7. Build overall site quality and authority, because Google indexes more from sites it trusts.

Should every page be indexed?

No. Pages such as internal search results, thank-you pages, filtered product lists, tag archives and login pages usually add no value in search results. Keeping them out of the index helps Google focus on your best content. A smaller index of strong pages often performs better than a large index full of weak ones (see Content Pruning).

Index coverage after a website change

After a redesign or Website Migration, watch the Pages report closely. A sudden rise in excluded pages, or important URLs disappearing, is an early warning that redirects, canonicals or robots rules have gone wrong.

Key Takeaways

  • Only indexed pages can appear in search results.
  • Google is selective and may crawl but not index weaker pages.
  • Use URL Inspection and the Pages report to diagnose problems.
  • Unique, helpful, well-linked pages are indexed most reliably.
  • Not every page should be indexed.

Related Terms

Curious how this applies to your website? Request a free SEO check-up, or keep learning in the SEO Encyclopedia.