Robots.txt

Robots.txt is a plain text file at the root of a website (for example, yourdomain.co.uk/robots.txt) that tells search engine crawlers which parts of the site they may or may not crawl. It is one of the first files Googlebot checks before Crawling a site.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

What a robots.txt file looks like

A simple robots.txt might contain:

  • User-agent: \* – the rules that follow apply to all crawlers.
  • Disallow: /wp-admin/ – do not crawl the admin area.
  • Allow: /wp-admin/admin-ajax.php – except this file, which the site needs.
  • Sitemap: https://www.yourdomain.co.uk/sitemap_index.xml – the location of your XML Sitemap.

You can add separate rules for specific crawlers, such as Googlebot, Bingbot or AI crawlers like GPTBot, by naming them in their own User-agent group.

What robots.txt is used for

  • Stopping crawlers wasting time on low-value areas, such as internal search results, carts, filters and admin pages.
  • Protecting server resources on very large sites (see Crawl Budget).
  • Pointing crawlers to your sitemap.
  • Choosing whether AI crawlers may use your content for training or AI search.

What robots.txt cannot do

  • It does not remove pages from Google. A blocked URL can still appear in results, usually without a description, if other pages link to it. To keep a page out of search results, use a Noindex Tag instead, and do not block that page in robots.txt, or Google will never see the noindex.
  • It is not security. Anyone can read your robots.txt, and badly behaved bots ignore it. Protect private content with passwords.
  • It does not stop indexing of content already known; it only stops crawling.

Common robots.txt mistakes

  1. Disallow: / left on after a site goes live, blocking the whole site. This happens surprisingly often after development.
  2. Blocking CSS and JavaScript files that Google needs for Rendering.
  3. Blocking pages you are trying to noindex or redirect, so Google cannot see the change.
  4. Typos in paths, since rules are case-sensitive.
  5. Using robots.txt to hide duplicate content instead of fixing it with canonical tags.

How to test robots.txt

Check the robots.txt report in Google Search Console, which shows the file Google last fetched and any errors. Use the URL Inspection tool to see whether a specific page is blocked. When editing the file, change one rule at a time and test important URLs before and after.

Robots.txt and AI crawlers

Many AI companies publish the names of their crawlers. You can allow or block them separately from search engines. Blocking AI crawlers may stop your content being used for training, but it can also reduce your visibility in AI answers. Decide based on your goals; see AI SEO for the trade-offs, and llms.txt for a newer, complementary file.

Robots.txt on WordPress

WordPress creates a virtual robots.txt automatically, and SEO plugins let you edit it. Always check the live file after installing a new plugin or changing the “discourage search engines” setting.

Further reading: Google’s introduction to robots.txt.

Key Takeaways

  • Robots.txt controls crawling, not indexing.
  • Use a noindex tag, not robots.txt, to keep a page out of search results.
  • Never block CSS or JavaScript that Google needs to render pages.
  • Check the live file after launches, migrations and plugin changes.
  • You can set separate rules for search engines and AI crawlers.

Related Terms

Want to know where your site stands? Book a free SEO health check, or keep learning in the SEO Encyclopedia.