AI Crawlers
AI crawlers are automated bots run by AI companies that fetch web pages, either to train language models, to build an index for AI search, or to read a page live when a user asks a question. Each identifies itself with a user agent name that you can manage in robots.txt.
How AI crawlers work
Like any web crawler, an AI crawler requests URLs, downloads the HTML and follows links to find more pages. The difference is what happens to the content afterwards. Most AI companies now run several separate bots, each with its own job:
- Training crawlers collect large amounts of public text used to train future models.
- Search or indexing crawlers build an index so the AI tool can find and cite sources when answering.
- User-triggered fetchers visit a specific page because a person asked the assistant to read it or look something up.
Main AI crawlers to know
| Company | User agents |
|---|---|
| OpenAI | GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user requests) |
| Anthropic | ClaudeBot (training), Claude-SearchBot (search), Claude-User (user requests) |
| Perplexity | PerplexityBot (search index), Perplexity-User (user requests) |
| Google-Extended (a robots.txt token controlling Gemini use, not a separate crawler) | |
| Common Crawl | CCBot (an open dataset used by many AI projects) |
Providers update these names from time to time, so check each company’s documentation before writing rules. You can read more about GPTBot and Google-Extended in their own entries.
Why AI crawlers matter for SEO
If an AI search crawler cannot reach your pages, the tool is far less likely to cite or link to you. On the other hand, some publishers do not want their work used for model training. Because training and search bots are separate, you can often allow one and block the other. Blocking a training crawler does not usually remove you from that company’s search answers, and vice versa.
How to manage AI crawlers
- Check your server logs to see which AI bots already visit. Log file analysis shows how often and which URLs.
- Decide your policy for each type: training, search and user-triggered.
- Add matching rules to your robots.txt for AI bots.
- If you use a CDN or firewall, check whether it blocks AI bots by default, as some now offer this setting.
- Review the rules every few months as new bots appear.
Common mistakes
- Blocking every AI bot without realising this also removes you from AI search citations.
- Assuming robots.txt is enforced. It is a request that reputable bots follow, not a security barrier.
- Trusting the user agent alone. Anyone can fake a name, so major providers publish IP ranges for verification.
- Forgetting that user-triggered fetchers may be treated differently from automatic crawlers by some providers.
Key Takeaways
- AI crawlers fetch pages for model training, AI search indexes or live user requests.
- Most AI companies run separate bots for each purpose, with different user agent names.
- You can allow AI search bots while blocking training bots in robots.txt.
- Robots.txt is a voluntary standard, so verify important bots by IP where possible.
- Review your AI crawler rules regularly because new bots appear often.
Related Terms
Browse more terms in the SEO Encyclopedia.