Crawl Budget
Crawl budget is the number of URLs Googlebot can and wants to crawl on your website in a given period. It is a combination of crawl capacity, how much crawling your server can handle, and crawl demand, how much Google wants to crawl your pages based on their importance and freshness.
Does crawl budget matter for your site?
For most small and medium websites with a few thousand pages or fewer, crawl budget is not a concern; Google can crawl everything comfortably. It becomes important for:
- Large sites with tens of thousands of URLs or more, such as ecommerce stores, marketplaces and publishers.
- Sites that generate many URLs automatically through filters, parameters or calendars.
- Sites with lots of new or frequently changing content.
- Sites with slow servers or frequent errors.
If new pages on a large site take weeks to be crawled, or many pages show “Discovered, currently not indexed”, crawl budget may be part of the problem. My guide to the Discovered currently not indexed fix covers this status.
The two parts of crawl budget
- Crawl capacity limit: Google crawls as fast as your server comfortably allows. Fast, reliable responses raise the limit; slow responses and 5xx errors lower it.
- Crawl demand: popular, frequently updated and well-linked pages are crawled more often. Pages Google considers low value are crawled less.
What wastes crawl budget
- Faceted navigation creating millions of filter combinations.
- URL parameters for sorting, tracking and session IDs.
- Duplicate pages and near-duplicates.
- Redirect chains and broken links.
- Soft 404s and thin pages.
- Infinite spaces, such as calendars with endless future dates.
- Hacked or spam pages created by attackers.
How to optimise crawl budget
- Block low-value URL patterns in Robots.txt, such as internal search results and endless filter combinations.
- Consolidate duplicates with redirects and canonical tags.
- Keep XML sitemaps clean, listing only important, indexable URLs with accurate lastmod dates.
- Improve server speed and fix 5xx errors.
- Strengthen Internal Linking to important pages so Google sees them as priorities.
- Remove or noindex thin content that adds little value (see Content Pruning).
- Fix redirect chains and broken links.
- Return 404 or 410 for removed content rather than redirecting everything.
How to monitor crawling
- The Crawl Stats report in Google Search Console shows total requests, response times, response codes and file types.
- Server log analysis shows exactly which URLs Googlebot visits and how often, revealing wasted crawling.
- Compare the number of URLs crawled with the number of pages you actually want indexed.
Crawl budget myths
Crawl budget is not a ranking factor. Being crawled more often does not make a page rank better; it simply means changes are picked up faster. Focus on making sure Google spends its crawling on the pages that matter.
Further reading: Google’s guide to managing crawl budget.
Key Takeaways
- Crawl budget is how many URLs Googlebot can and wants to crawl on your site.
- It mainly matters for large sites with many URLs or frequent changes.
- Filters, parameters, duplicates and redirect chains waste crawling.
- Clean sitemaps, fast servers and strong internal links help Google focus.
- Being crawled more often is not a ranking factor in itself.
Related Terms
Want help putting this into practice? Get a free SEO check, or keep learning in the SEO Encyclopedia.