GPTBot

GPTBot is OpenAI’s web crawler, used to collect publicly available web content that may be used to train its AI models. Site owners can allow or block it in robots.txt, and blocking it is separate from controlling whether your pages appear in ChatGPT search.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

What GPTBot does

OpenAI announced GPTBot in August 2023 and published instructions for blocking it. The bot identifies itself with "GPTBot" in its user agent string. OpenAI says it filters out sources that are behind paywalls, known to collect personal information, or that break its policies.

GPTBot is a training crawler. Its job is to gather text that could improve future models, not to fetch sources for a specific answer.

GPTBot vs OpenAI’s other bots

OpenAI runs more than one crawler, and each can be controlled on its own:

  • GPTBot: collects content for model training.
  • OAI-SearchBot: finds and indexes pages so they can be shown and linked in ChatGPT search.
  • ChatGPT-User: visits pages when a user’s request in ChatGPT needs to fetch a specific URL.

OpenAI’s documentation says sites that opt out of GPTBot can still appear in ChatGPT search results, as long as OAI-SearchBot is allowed. This means blocking GPTBot is mainly a decision about training, not about visibility in ChatGPT answers.

How to block or allow GPTBot

You control GPTBot through your robots.txt file. To block it from the whole site, add:

User-agent: GPTBot

Disallow: /

To block only certain folders, list them with separate Disallow lines. To keep ChatGPT search visibility while opting out of training, block GPTBot but leave OAI-SearchBot allowed. Changes can take a while to be picked up, because bots re-read robots.txt periodically.

Should you block GPTBot?

There is no single right answer. Points to weigh up include:

  • Reasons to allow: your brand, facts and expertise may be better represented in future models.
  • Reasons to block: you may not want original or paid content used to train commercial AI without permission, or you may be negotiating AI content licensing.
  • Server load: heavy crawling can use bandwidth, though most sites will not notice.

Many large news publishers chose to block GPTBot after it launched, while many businesses that want brand exposure leave it allowed.

How to check GPTBot visits

  1. Search your server logs for the GPTBot user agent.
  2. Verify visits against the IP ranges OpenAI publishes, since user agents can be faked.
  3. Test your rules with a robots.txt checker to confirm they match the right bot.

Common mistakes

  • Blocking OAI-SearchBot as well as GPTBot when you only meant to opt out of training, which can remove you from ChatGPT search.
  • Expecting a block to remove content already collected. It only affects future crawling.
  • Using a typo or the wrong case in the user agent name, so the rule never matches.
  • Relying on robots.txt alone to protect sensitive content. Anything private should sit behind a login.

Further reading: OpenAI’s documentation on its crawlers.

Key Takeaways

  • GPTBot is OpenAI’s crawler for collecting model training data.
  • OAI-SearchBot, not GPTBot, controls whether pages can appear in ChatGPT search.
  • You can block GPTBot in robots.txt while still allowing ChatGPT search visibility.
  • Verify GPTBot visits using OpenAI’s published IP ranges.
  • Whether to block it depends on how you value training exposure against control of your content.

Related Terms

Browse more terms in the SEO Encyclopedia.