Crawl Errors

Crawl errors are problems that stop a search engine crawler from successfully fetching a page or a whole website. They range from site-wide issues such as DNS or server failures to page-level issues such as broken links returning 404 errors, and serious ones can stop pages being indexed.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

Types of crawl errors

Crawl errors fall into two broad groups.

Site-level errors

  • DNS errors: the crawler cannot look up your domain, so it cannot reach the server at all.
  • Server connectivity errors: the server is down, too slow or refuses the connection.
  • Robots.txt fetch errors: if robots.txt returns a server error, Google may pause crawling the site until it can read the file.

URL-level errors

  • Not found: a 404 error means the page does not exist.
  • Soft 404s: a page returns a 200 OK code but looks empty or like an error page; see soft 404.
  • Server errors: 5xx server errors on specific URLs, often caused by bugs or timeouts.
  • Redirect errors: chains that are too long, loops, or redirects to broken URLs.
  • Access denied: 401 or 403 responses blocking the crawler.

Why crawl errors matter

A page that cannot be fetched cannot be indexed, and if it is already indexed it may eventually drop out. Site-level errors are the most urgent because they affect everything. Persistent server errors can also cause Google to slow down its crawling to avoid overloading your server, which delays discovery of new content.

Not every error is a problem, though. Google has said that 404s on pages that genuinely no longer exist are normal and do not harm the rest of your site.

How to find crawl errors

  1. Open the Page indexing report in Google Search Console to see URLs not indexed and the reasons.
  2. Check the Crawl stats report (under Settings) for host status problems with DNS, server connectivity and robots.txt.
  3. Inspect individual URLs with the URL Inspection tool to see the last crawl result.
  4. Run a site crawl with an SEO crawler to catch broken internal links and redirect problems.
  5. Review server logs for error responses served to bots, using log file analysis.

How to fix common crawl errors

  • Fix DNS or hosting issues with your provider first, as they block everything else.
  • Redirect removed pages that have a clear replacement using a 301 redirect, and let others return 404 or 410.
  • Update internal links that point to broken or redirected URLs.
  • Shorten or remove redirect chains and loops.
  • Fix soft 404s by returning a proper 404 code or adding real content.
  • Investigate repeated 5xx errors with your developer or host, especially on busy templates.

Common mistakes

  • Redirecting every 404 to the homepage, which Google may treat as a soft 404 anyway.
  • Panicking over old 404s from long-deleted pages that nobody links to.
  • Ignoring host status warnings because individual pages still seem to load in a browser.

Key Takeaways

  • Crawl errors stop search engines from fetching your pages.
  • Site-level errors like DNS and server failures are the most urgent.
  • Search Console’s Page indexing and Crawl stats reports show most crawl errors.
  • 404s for pages that no longer exist are normal and not harmful.
  • Fix broken internal links and redirect chains to help crawlers.

Related Terms

Browse more terms in the SEO Encyclopedia.