Robots.txt for AI Bots

Robots.txt for AI bots means using your site’s robots.txt file to tell AI crawlers which pages they may or may not fetch. Because most AI companies run separate bots for training, search and user requests, careful rules let you opt out of training while staying visible in AI answers.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

How it works

A robots.txt file sits at the root of your domain and contains groups of rules. Each group starts with a User-agent line naming a bot, followed by Allow or Disallow lines for paths. Reputable AI crawlers read this file before crawling and follow the group that matches their name. If no group matches, they follow the User-agent: * group.

Common AI user agents

  • Training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google’s token for Gemini), Applebot-Extended (Apple’s token for AI training).
  • AI search: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity).
  • User-triggered: ChatGPT-User, Claude-User and Perplexity-User, which fetch a page because a person asked.

Names change and new bots appear, so always confirm against each provider’s own documentation. Tokens are matched without regard to case, but spelling must be exact.

Example: block training, allow AI search

This setup opts out of model training while keeping your pages available for AI search citations:

User-agent: GPTBot

User-agent: ClaudeBot

User-agent: Google-Extended

User-agent: CCBot

Disallow: /

Several User-agent lines can share one set of rules. Bots not named here, including OAI-SearchBot and Googlebot, follow your general rules instead. Blocking Google-Extended does not affect Google Search.

How to set it up

  1. Decide your policy for training, AI search and user-triggered bots separately.
  2. Check your logs to see which bots actually visit.
  3. Write a group for each bot or set of bots, using exact user agent tokens.
  4. Test the file with a robots.txt validator and confirm it is served at /robots.txt with a 200 status.
  5. Review it every few months, since providers add and rename bots.

If you run several sites or subdomains, remember that each host needs its own robots.txt file. Rules on www do not apply to a shop or blog subdomain.

Limits and common mistakes

  • It is voluntary. Robots.txt is a request, not access control. Bots that ignore it need blocking at server or CDN level.
  • Some fetchers behave differently. Certain providers say user-triggered fetches may not follow robots.txt in the same way, because a human made the request.
  • It does not remove existing data. Blocking a bot today does not delete content already collected.
  • Blanket blocks have side effects. Using User-agent: * with Disallow: / blocks search engines too.
  • Robots.txt does not stop indexing. A blocked URL can still be listed if linked elsewhere. Use a noindex tag to keep pages out of search indexes.

Key Takeaways

  • Robots.txt lets you set separate rules for each AI bot by user agent.
  • You can block training bots while still allowing AI search bots to cite you.
  • Always use exact user agent tokens from each provider’s documentation.
  • Robots.txt is voluntary and does not delete data already collected.
  • Test your file after every change so you never block search engines by accident.

Related Terms

Not sure how your own site measures up? Request a free SEO review, or keep learning in the SEO Encyclopedia.