User-Agent

A user-agent is a short piece of text that a browser, app or bot sends with every request to tell a web server what it is. In SEO, user-agents matter because they identify search engine and AI crawlers, and they are what robots.txt rules use to target specific bots.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

What a user-agent looks like

Every time a browser or crawler asks for a page, it sends an HTTP header called User-Agent. A typical browser string names the browser, its version, the rendering engine and the operating system. Crawlers include their own name, often with a link to information about the bot.

Google’s smartphone crawler, for example, sends a string that looks like a mobile Chrome browser and includes the token Googlebot. Bing’s crawler includes bingbot, and OpenAI’s training crawler includes GPTBot.

User-agents in robots.txt

In robots.txt, each group of rules starts with a User-agent: line naming the crawler it applies to. For example, User-agent: GPTBot followed by Disallow: / asks OpenAI’s training crawler not to fetch any pages. User-agent: * applies to all crawlers that do not have a more specific group.

Some tokens control use rather than crawling. Google-Extended is a robots.txt token that lets sites opt out of their content being used for some Google AI models, without affecting Google Search crawling.

Rules for AI bots are covered in more depth in robots.txt for AI bots.

How SEOs use user-agents

  • Log analysis: server logs record the user-agent of each request, so log file analysis can show how often Googlebot visits and what it requests.
  • Crawler testing: SEO tools let you crawl a site as a mobile or desktop Googlebot to see what it would see.
  • Debugging: browser developer tools can switch user-agent to check how a site responds to different devices.
  • Bot management: security tools and firewalls use user-agents, along with other checks, to allow or block bots.

Fake user-agents and verification

A user-agent is easy to fake: any script can claim to be Googlebot. Scrapers often do this to slip past blocks. To verify a genuine Google crawler:

  1. Take the IP address from your server log.
  2. Run a reverse DNS lookup and check it resolves to a googlebot.com, google.com or googleusercontent.com host name.
  3. Run a forward DNS lookup on that host name and confirm it returns the same IP.
  4. Alternatively, check the IP against the lists of crawler IP ranges that Google publishes.

Bing and other major search engines offer similar verification methods.

Common mistakes

  • Serving different content to search engine user-agents than to people, which is cloaking and breaks Google’s spam policies.
  • Blocking crawlers by user-agent on a firewall or CDN without realising it, causing pages to drop from search.
  • Misspelling a crawler name in robots.txt, so the rule silently does nothing.
  • Blocking all unfamiliar bots and accidentally shutting out legitimate search or AI crawlers.

Key Takeaways

  • A user-agent is the text a browser or bot sends to identify itself.
  • Robots.txt rules target crawlers by their user-agent token.
  • Log file analysis uses user-agents to show how bots crawl your site.
  • User-agents can be faked, so verify important bots with DNS lookups.
  • Showing search bots different content from users counts as cloaking.

Related Terms

Wondering what to fix first on your site? Get a free check of your site’s SEO, or keep learning in the SEO Encyclopedia.