Crawling
Crawling is the process search engines use to discover and download pages on the web. Google’s crawler, called Googlebot, visits URLs, fetches their content and follows the links it finds to discover more pages. A page that is never crawled cannot be indexed or ranked.
How Google discovers URLs
Googlebot finds new pages in several ways:
- Following links from pages it already knows, both on your site (Internal Linking) and from other websites (Backlinks).
- Reading XML sitemaps that list the URLs you want crawled.
- URL Inspection requests submitted in Google Search Console.
- Redirects pointing from old URLs to new ones.
Links are the most important route. A page with no internal links pointing to it, known as an orphan page, is easy for Google to miss or to treat as unimportant.
What happens during a crawl
When Googlebot visits a URL, it first checks your Robots.txt file to see whether it is allowed to fetch it. It then requests the page from your server, receives an HTTP status code and the HTML, and queues any links it finds. Pages that rely on JavaScript may be sent for Rendering before Google sees the full content. The fetched content is then passed on for Indexing.
How often Google crawls
Crawl frequency depends on how important and how frequently updated your pages appear to be. A busy news homepage may be crawled many times a day, while a rarely updated page deep in a small site may only be revisited every few weeks. Google also adjusts its crawl rate so it does not overload your server. For very large sites, the number of pages Google is willing to crawl is known as the Crawl Budget.
What stops pages being crawled
- Robots.txt disallow rules blocking a folder or the whole site.
- Server errors (5xx codes) or very slow responses, which make Google crawl less.
- Login walls and pages only reachable through forms or site search.
- Links that are not real links, such as JavaScript click events without an href.
- Endless URL variations from filters and parameters wasting crawl activity (Faceted Navigation).
- Broken internal links pointing to 404 pages.
How to check crawling
- Use the Crawl Stats report in Search Console to see how many requests Googlebot makes, response times and errors.
- Use the URL Inspection tool to see when a page was last crawled.
- Crawl your own site with an SEO crawler to see what a bot can reach from the homepage.
- For deeper insight, analyse your server logs (Log File Analysis).
How to improve crawling
- Link to every important page from relevant pages, ideally within three clicks of the homepage.
- Keep your XML sitemap clean and up to date with only indexable URLs.
- Fix broken links and long Redirect Chains.
- Make sure your server responds quickly and reliably.
- Block genuinely useless URL patterns, such as internal search results, in robots.txt.
- Use normal HTML links with href attributes.
Remember that crawling is not the same as indexing. Google can crawl a page and still decide not to include it in search results. For that next step, see Indexing, and for a beginner overview read how to get your website on Google.
Key Takeaways
- Crawling is how search engines discover and fetch pages.
- Internal links, sitemaps and backlinks are the main discovery routes.
- Robots.txt blocks, server errors and non-HTML links stop crawling.
- Use Crawl Stats and URL Inspection in Search Console to check activity.
- Crawling does not guarantee indexing.
Related Terms
Need a second opinion on your website? Ask for a free website SEO check, or keep learning in the SEO Encyclopedia.