ClaudeBot is Anthropic's web crawler, identified by the User-Agent strings 'ClaudeBot' and 'anthropic-ai'. It indexes publicly accessible pages to power Claude's search, citation, and (with opt-in) model training corpora.
Allowing ClaudeBot is essential for citation inside Claude. Claude is one of the three dominant AI engines in 2026 alongside ChatGPT and Perplexity, with strong adoption in technical and professional contexts.
Anthropic publishes crawler documentation and IP ranges. The crawler respects robots.txt. A separate user agent 'ClaudeUser' fetches pages on behalf of individual Claude users during conversations.
Read next
Frequently asked
What's the difference between ClaudeBot and ClaudeUser?
ClaudeBot is the background crawler that indexes the web. ClaudeUser is the user-fetch agent that retrieves specific pages during a Claude conversation when a user asks Claude to read a URL. Allow both for full access.
Does Claude use the same index as ChatGPT?
No. Anthropic and OpenAI maintain independent indexes. Citation behavior, ranking signals, and snapshot freshness differ between the two engines.
How do I allow ClaudeBot in robots.txt?
Add a User-agent: ClaudeBot block with Allow: /. Also add separate blocks for anthropic-ai and ClaudeUser for full coverage.
More from AIO and AEO
All termsAI Visibility
How often and how accurately a brand appears inside AI engine answers (ChatGPT, Claude, Perplexity).
ReadGPTBot
OpenAI's web crawler that indexes content for ChatGPT and the OpenAI training corpus.
ReadPerplexityBot
Perplexity AI's web crawler that powers citation in Perplexity's search and answer engine.
Read