Googlebot

Googlebot is the name of Google’s web crawler: the automated program that visits web pages, downloads their content and passes it to Google’s systems for indexing. If Googlebot cannot reach or render a page, that page is unlikely to appear in Google Search.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

What Googlebot does

Googlebot finds new and updated pages by following links, reading XML sitemaps and revisiting URLs it already knows. For each URL it requests the page from your server, much as a browser does, and records the response.

Pages are then queued for rendering, where Google runs the page’s JavaScript in an up-to-date version of Chromium so it can see content added by scripts.

The rendered result is what Google uses for indexing. Rendering can happen some time after the first crawl, so content that relies heavily on JavaScript may take longer to be indexed than content delivered in plain HTML.

Googlebot does not crawl every page equally. Important, frequently updated pages tend to be revisited often, while low-value or rarely changing URLs may be crawled far less. On very large sites, managing which URLs get crawled becomes a real concern.

Types of Googlebot

Google uses several crawlers, each identifying itself with a user agent string:

  • Googlebot Smartphone: the main crawler for most sites since Google moved to mobile-first indexing.
  • Googlebot Desktop: still used for some crawling.
  • Googlebot Image, Googlebot Video and Googlebot News: specialist crawlers for media and news content.
  • Other Google crawlers: such as Storebot for shopping and AdsBot for checking ad landing pages. Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used for Gemini models.

How to control Googlebot

  1. Use robots.txt to stop Googlebot crawling sections of a site, such as internal search results.
  2. Use a noindex directive to keep a crawlable page out of the index. Googlebot must be able to crawl the page to see the noindex.
  3. Keep server responses fast and reliable. Google has said it slows crawling when a server responds slowly or returns many errors.
  4. Use internal links and sitemaps to point Googlebot at the pages that matter most.

How to check Googlebot activity

The Crawl Stats report in Google Search Console shows how often Googlebot visits, which responses it gets and how quickly your server replies. The URL Inspection tool shows how Googlebot sees an individual page, including the rendered HTML.

For the most detail, log file analysis reveals every request Googlebot makes, which helps you spot wasted crawling on unimportant URLs.

Fake Googlebots

Anyone can claim to be Googlebot in a user agent string, and scrapers often do. Google recommends verifying a visitor with a reverse DNS lookup: the hostname should end in googlebot.com, google.com or googleusercontent.com, and a forward DNS lookup on that hostname should return the original IP address. Google also publishes lists of the IP ranges its crawlers use.

Further reading: Google’s Googlebot documentation.

Key Takeaways

  • Googlebot is Google’s crawler, which fetches pages for indexing.
  • Most sites are crawled mainly by Googlebot Smartphone.
  • Googlebot renders JavaScript, but rendering can happen after the initial crawl.
  • robots.txt controls crawling, while noindex controls indexing.
  • Check Googlebot activity in the Crawl Stats report and your server logs.
  • Verify genuine Googlebot with reverse and forward DNS lookups.

Related Terms

Not sure how your own site measures up? Request a free SEO review, or keep learning in the SEO Encyclopedia.