Glossary

What are AI crawlers?

The bots AI companies use to read the web: which one trains models, which one feeds live answers, and how each one is controlled from robots.txt.

Definition by Apoda, GEO platform Updated October 9, 2026 Full glossary

AI crawlers are the programs artificial intelligence companies use to read web pages. They don't all do the same thing: some collect content to train models, others build the search index an assistant consults when answering, and others fetch a specific page because a user asked for it right then. Blocking the wrong one takes you out of the answers without you noticing.

What's the difference between training, indexing and fetching a page?

It's the distinction that costs the most money when it's confused.

Training. The bot collects text that will be used to train a future model. Blocking it doesn't affect whether the assistant mentions you today.
Indexing for search. The bot builds the index the assistant consults when it needs current information. Blocking it does take you out of answers with citations.
Fetching a page live. The bot goes to a specific URL because a user or a task asked for it. Blocking it means the assistant can't open your page when someone hands it over.

Many people block all three at once, thinking they're only preventing training.

What are the main AI crawlers?

Bots from the main AI companies. Last reviewed: October 9, 2026.
Bot	Company	What it's used for	How it's controlled
`GPTBot`	OpenAI	Collects pages to train models. It doesn't feed ChatGPT's live answers.	robots.txt, its own entry for GPTBot.
`OAI-SearchBot`	OpenAI	Builds the search index ChatGPT consults when answering. Blocking it takes you out of the results with citations.	robots.txt, its own entry for OAI-SearchBot.
`ChatGPT-User`	OpenAI	Fetches a page on the spot because a user or a task asked for it. It doesn't index or train.	robots.txt, its own entry for ChatGPT-User.
`ClaudeBot`	Anthropic	Collects pages for training.	robots.txt, its own entry for ClaudeBot.
`Claude-User`	Anthropic	Fetches a page on the spot when a Claude user asks for it.	robots.txt, its own entry for Claude-User.
`Claude-SearchBot`	Anthropic	Indexes pages to improve the search results Claude uses when answering.	robots.txt, its own entry for Claude-SearchBot.
`PerplexityBot`	Perplexity	Indexes pages so they can appear cited in Perplexity's answers.	robots.txt, its own entry for PerplexityBot.
`Google-Extended`	Google	It isn't a crawler: it's a control. It decides whether the content Googlebot already fetched can be used to train Gemini and feed its answers.	robots.txt. Blocking it doesn't affect your position in Google Search.
`Applebot-Extended`	Apple	Not a crawler either: it controls whether what Applebot already fetched can be used to train Apple's models.	robots.txt. Blocking it doesn't affect Siri or Spotlight.
`meta-externalagent`	Meta	Collects pages to train Meta's models and for its AI products.	robots.txt, its own entry for meta-externalagent.
`Bravebot`	Brave	Feeds the Brave Search index, which in turn feeds Claude's web search. It's the least-known bot on the list and the one most people leave out without realizing.	A special case: see the note below the table.

This table ages fast: AI companies add, rename and retire user agents several times a year. The date above says when it was last checked against each company's official documentation.

Why is Brave on this list?

Because the Brave Search index feeds Claude's web search, and almost nobody takes it into account when reviewing their robots.txt.

Anthropic hasn't officially named its search provider, but it included Brave Search in its public list of subprocessors under the web search category, and several independent analyses found that Claude and Brave return the same citations for the same query. The evidence is strong, even if it isn't a direct confirmation.

The practical consequence is what matters: if you aren't visible to Brave, you're probably not citable by Claude, even if ClaudeBot is perfectly allowed in your robots.txt.

Sources: subprocessor list in Anthropic's Trust Center; TechCrunch, “Anthropic appears to be using Brave to power web search for its Claude chatbot”, March 2025.

How is Brave's crawler controlled?

It's the odd case on the list, and it's worth understanding before touching anything.

Brave documents that its crawler doesn't announce a distinct user agent, precisely so it isn't discriminated against by sites that only let Google through. From that follows the rule almost nobody knows: if Googlebot can't crawl a page, Brave's bot won't either.

That flips the usual advice. Adding a line for a Brave bot to your robots.txt does nothing; what you need to check is that you aren't blocking Googlebot in the sections you want AI to be able to cite. Closed directories, staging environments with Disallow: /, and blocks inherited from a migration are the most common causes.

Some bot directories report a Bravebot user agent. It's worth including in robots.txt just in case — it costs one line — but without relying on it: Brave's documentation still says effective control goes through Googlebot's access.

Source: Brave Search help center, “Brave Search Crawler”.

What's worth checking on a site?

Four things, in this order:

That robots.txt doesn't block Googlebot in the sections you want AI to read. This point alone explains part of the cases where a business doesn't show up in any answer.
That the indexing bots are allowed, even if you decide to block the training ones. They're separate entries: OAI-SearchBot is not GPTBot.
That there are no blocks outside robots.txt. A firewall, a WAF or a Cloudflare rule can be rejecting bots that robots.txt allows. You can't see this without looking at the server logs.
That critical pages don't depend on JavaScript to show their content. Several of these bots don't run JavaScript: what renders in the browser may not exist for them.