AI search guide · Claude
How Claude finds and cites sources.
What Anthropic documents about Claude’s crawlers and web search, which bots to let in, and what checked studies say about its answers.
Zach ChmaelChecked September 30, 2026 · from Anthropic’s documentation and checked studies
Free AI visibility check
The short answer
Claude, in brief.
Claude can search the web and cite what it finds, and Anthropic runs three bots with separate jobs: Claude-SearchBot indexes pages for search, Claude-User fetches a page when someone asks, and ClaudeBot collects training data. Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude’s answers; ClaudeBot is its training crawler. Anthropic’s web search documentation doesn’t name the search index behind it.
Sources
Where it finds sources
Anthropic runs Claude-SearchBot to improve search results for Claude’s users: it analyzes online content to make search responses more relevant and accurate.
Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? · read September 30, 2026When someone asks Claude a question, it may visit websites with its Claude-User agent to retrieve content for the answer.
Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? · read September 30, 2026When Claude searches the web through Anthropic’s API, citations are always on: answers link to the pages they drew on.
Anthropic, Web search tool · read September 30, 2026Anthropic’s web search tool documentation doesn’t name the search index behind it.
Anthropic, Web search tool · read September 30, 2026
Crawlers
The crawlers to know
| User agent | What it does | robots.txt |
|---|---|---|
Claude-SearchBot | Indexes pages to improve Claude’s search results | Obeys it. Blocking it may reduce your visibility in Claude’s search results, Anthropic says. |
Claude-User | Fetches a page when a user’s question needs it | Obeys it. Blocking it stops Claude retrieving your pages for users’ questions. |
ClaudeBot | Collects content that may be used to train Anthropic’s models | Obeys it. Blocking it signals that your future content should be left out of training, Anthropic says. |
Source: Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? · read September 30, 2026
To be found by Claude without being trained on, allow Claude-SearchBot and Claude-User and disallow ClaudeBot. Anthropic asks for the rules in each subdomain’s robots.txt, and says blocking its IP addresses may not work.
Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? · read September 30, 2026Citations
What it tends to cite
No checked study isolates Claude’s citations yet. This one pools eight engines, Claude among them; the line under the number says what was measured.
Stability
How much its answers change
No checked study covers this engine on its own yet. The statistics page’s reliability section has the cross-engine figures.
First steps
What to fix first
- 01
Allow Claude-SearchBot and Claude-User in robots.txt, and decide on ClaudeBot separately.
- 02
Set the rules on every subdomain: Anthropic asks for them in each subdomain’s own robots.txt.
- 03
Put what matters in the HTML: ClaudeBot didn’t run JavaScript in Vercel and MERJ’s test of the major AI crawlers.
- 04
Watch your server logs for Claude-User: its visits show which pages people’s questions pull, and a crawl is not the same as being cited.
- 05
Earn mentions off your own site: in AirOps’ B2B test, which included Claude, 85% of brand mentions came from third-party sites.
Your company
See what AI says about you
Trovance sweeps Claude for your buyer questions every week on Growth and Agency. See the plans
Go deeper
Read next
- How to Measure AI Agent Traffic in Server LogsThe fetch layer is a server-side measurement, and most teams are reading the wrong instrument.Agentic Marketing11 min read
- How to Track AI Citations Across ChatGPT, Perplexity, and ClaudeOne answer is noise. A frozen question set, a few phrasings run weekly on every engine, and a log that separates mention from citation turn it into a measurement.SEO/GEO11 min read
- Blocking AI Crawlers Is a Policy Choice, Not a Citation SwitchA purpose-specific worksheet for service businesses choosing search access, training preferences, and user-requested visits.GEO11 min read
- Allowing Googlebot does not allow GPTBotSeparate user agents, separate permissions. Here is the current crawler list, the copy-paste templates, and the honest ceiling on what they buy.How-to11 min read
FAQ
Questions, answered.
How do I get my company into Claude’s answers?
Let Claude-SearchBot and Claude-User reach your pages, keep the facts that matter in the HTML, and earn coverage on the sites answers draw on. Then check what Claude says, more than once: answers vary from run to run.
Does blocking ClaudeBot remove me from Claude?
No. ClaudeBot collects training data; blocking it signals that your future content should be left out of training. Claude-SearchBot and Claude-User are the bots that affect whether Claude can find and retrieve your pages.
Which search engine does Claude use?
Anthropic’s web search documentation doesn’t name one. Its crawler article does say Claude-SearchBot analyzes online content to improve the relevance and accuracy of Claude’s search results.
Rather ask a person? Text us.
Get understood. Get cited. Get chosen.
Text Trovance
Or, in iMessage, send this note to +1 (839) 205-9508:
Hi Trovance, show me how AI sees my company.
You choose to send it. Trovance replies, and sends the updates you ask for, over iMessage through our messaging provider. Reply STOP anytime. Tapping or sending never bills you.