AI crawler checker
Which AI bots can read your site?
Enter your domain. Trovance reads your robots.txt and shows, for each of 26 AI and search bots, whether it may read your page and the exact rule that decides it.
It reads one public file, your robots.txt, and nothing else. Free, no sign-up.
Limits
What robots.txt can’t show
robots.txt says what a bot may fetch. These decide the rest, and none of them is in the file.
A firewall or CDN can turn bots away first
Cloudflare, for one, lets a site block AI bots by purpose (search, agent or training) or AI crawlers one by one, and custom WAF rules go further. Google’s own advice for its AI features is to make sure crawling is allowed in robots.txt and by any CDN or hosting infrastructure.
Sources: Cloudflare, Block AI Bots · read October 9, 2026; Cloudflare, Manage AI crawlers · read October 9, 2026; Google Search Central, AI Features and Your Website · read October 9, 2026
Search Console’s Search generative AI control
It decides whether your links and content can appear in AI Overviews, AI Mode and generative AI features in Discover. It’s on by default and set in Search Console, so robots.txt never shows it.
Source: Search Console Help, Search generative AI control · read October 9, 2026
Googlebot crawls for Search and its AI features alike
Google says robots.txt rules for Googlebot are the control for how Search crawls your site, AI features included, so robots.txt can’t keep a page in Search but out of AI Overviews. To limit what’s shown, Google points to nosnippet, data-nosnippet, max-snippet and noindex.
Source: Google Search Central, AI Features and Your Website · read October 9, 2026
Google-Extended isn’t the AI Overviews switch
Google says it governs whether content it crawls may train Gemini models and ground answers in Gemini Apps and Vertex AI, and that it doesn’t affect inclusion in Google Search.
Source: Google Search Central, Google’s common crawlers · read October 9, 2026
An allowed page can still be out of reach
A page behind a login, or one whose words only appear once JavaScript runs, can be allowed and still unread by a bot that doesn’t sign in or run scripts.
Each bot can get a different robots.txt
A server can serve each bot a file of its own, chosen by its user agent or by its verified IP addresses. The checker reads the one served to its own user agent, TrovanceRobotsCheck, which may not be the one an AI bot gets.
Directory
The AI bot directory
All 26 bots the checker reads, each from its maker’s own documentation: what it does, what the maker says it does with robots.txt, and when that page was read.
| Bot | What it does | Follows robots.txt? | Maker’s page |
|---|---|---|---|
| Answer bots Fetch a page when someone asks an assistant. | |||
ChatGPT-UserOpenAI | Visits a page when a user’s request in ChatGPT or a custom GPT needs it | Not alwaysMay not apply, since a user started the visit; it doesn’t decide what appears in search. | Overview of OpenAI Crawlersread October 9, 2026 |
Claude-UserAnthropic | Fetches a page when a user’s question needs it | YesObeys it. Blocking it stops Claude retrieving your pages for users’ questions. | Does Anthropic crawl data from the web, and how can site owners block the crawler?read October 9, 2026 |
Perplexity-UserPerplexity | Visits a page when a user’s question needs it, and may link it in the answer | Not alwaysGenerally ignores it, since a user asked for the visit. | Perplexity Crawlersread October 9, 2026 |
Meta-ExternalFetcherMeta | Fetches individual links at a user’s request, including to help Meta’s AI complete tasks for them | Not alwaysMay bypass it, since the fetch is at a user’s request. | Meta Web Crawlersread October 9, 2026 |
Amzn-UserAmazon | Fetches live information from the web to answer a customer’s question, such as one asked of Alexa | Not alwaysMay not follow all of it, since a user can start the fetch. | About AmazonBotread October 9, 2026 |
MistralAI-UserMistral AI | Visits a page when a user asks Mistral’s Vibe a question, and links the source in its answer | YesMistral says this robots.txt token governs which sites these user requests can reach. | Mistral crawlersread October 9, 2026 |
DuckAssistBotDuckDuckGo | Crawls pages in real time for DuckDuckGo’s AI-assisted answers, which cite their sources | YesObeys it. A disallow takes effect after 72 hours and doesn’t change your place in DuckDuckGo’s search results. | Is DuckAssistBot related to DuckDuckGo?read October 9, 2026 |
| Search bots Index pages for a search product. | |||
OAI-SearchBotOpenAI | Surfaces sites in ChatGPT’s search features | YesObeys it. Block it and your pages aren’t shown in ChatGPT’s search answers, though they can still appear as navigational links. | Overview of OpenAI Crawlersread October 9, 2026 |
Claude-SearchBotAnthropic | Indexes pages to improve Claude’s search results | YesObeys it. Blocking it may reduce your visibility in Claude’s search results, Anthropic says. | Does Anthropic crawl data from the web, and how can site owners block the crawler?read October 9, 2026 |
PerplexityBotPerplexity | Surfaces and links websites in Perplexity’s search results; not used for training | YesObeys it. Perplexity recommends allowing it, and its published IP ranges. | Perplexity Crawlersread October 9, 2026 |
GooglebotGoogle | Crawls for Google Search, including its AI features | YesObeys it. Googlebot’s rules are the control for how Search crawls your site. | AI Features and Your Websiteread October 9, 2026 |
BingbotMicrosoft | Crawls for Bing Search, whose crawling and indexing Bing says Copilot’s search experiences rely on | YesObeys it. Microsoft says robots.txt controls crawl access, not indexing, and lists blocking Bingbot as something to avoid. | Bing Webmaster Guidelinesread October 9, 2026 |
ApplebotApple | Crawls for search in Spotlight, Siri and Safari; its data may add context to AI answers, and help train Apple’s models unless you disallow Applebot-Extended | YesObeys it. With no Applebot rules it follows your Googlebot rules, and it ignores crawl-delay. | About Applebotread October 9, 2026 |
Meta-WebIndexerMeta | Crawls the web to improve the quality of Meta AI’s search results | YesMeta says a robots.txt disallow blocks it, and allowing it helps Meta AI cite and link to your content. | Meta Web Crawlersread October 9, 2026 |
Amzn-SearchBotAmazon | Improves search in Amazon products and services, including Alexa; not used for generative AI training | YesObeys it. With no rules of its own it follows the rules you give other search bots. | About AmazonBotread October 9, 2026 |
MistralAI-IndexMistral AI | Indexes pages for Mistral’s search, which helps answer questions in Vibe; not used for training | Doesn’t sayMistral lists it among its robots.txt tags but doesn’t describe how it treats the rules. | Mistral crawlersread October 9, 2026 |
BytespiderByteDance | ByteDance’s crawler for Toutiao Search; its page doesn’t mention AI or training | Doesn’t sayByteDance’s Toutiao Search docs explain how its crawler matches robots.txt rules, like Google; its Bytespider page doesn’t say outright that it obeys them. | About Bytespider (in Chinese), Toutiao Search webmaster platformread October 9, 2026 |
| Training bots Collect data for model training. | |||
GPTBotOpenAI | Collects content that may be used to train OpenAI’s models | YesObeys it. Blocking it opts you out of training, not out of search. | Overview of OpenAI Crawlersread October 9, 2026 |
ClaudeBotAnthropic | Collects content that may be used to train Anthropic’s models | YesObeys it. Blocking it signals that your future content should be left out of training, Anthropic says. | Does Anthropic crawl data from the web, and how can site owners block the crawler?read October 9, 2026 |
Meta-ExternalAgentMeta | Crawls the web for uses such as training Meta’s foundation AI models, or improving products by indexing content | YesMeta says a robots.txt disallow blocks it; allow up to 24 hours for a change to take effect. | Meta Web Crawlersread October 9, 2026 |
AmazonbotAmazon | Improves Amazon’s products and services; may be used to train Amazon’s AI models | YesObeys it, but not crawl-delay. Amazon says changes can take about 24 hours to show. | About AmazonBotread October 9, 2026 |
MistralAI-TrainingMistral AI | Crawls web content to help build datasets for training Mistral’s generative AI models; not used for search indexing or to answer live user queries in Vibe | YesMistral says webmasters can disallow it in their robots.txt file. | Mistral crawlersread October 9, 2026 |
| Control tokens Govern how your content is used. Not crawlers. | |||
Google-ExtendedGoogle | A robots.txt token, not a crawler: governs Gemini training and grounding | YesDisallowing it tells Google not to use content it crawls from your site for Gemini training or grounding; it doesn’t affect Search. | Google’s common crawlersread October 9, 2026 |
Applebot-ExtendedApple | A robots.txt token, not a crawler: decides whether Apple trains its foundation models on what Applebot crawled | YesDisallow it to opt out of training; your pages can still appear in Apple’s search results. | About Applebotread October 9, 2026 |
| Other bots None of those: an ad-page check, an open web archive. | |||
OAI-AdsBotOpenAI | Validates the safety of web pages submitted as ads on ChatGPT, and may use their content to decide when to show the ad | Doesn’t sayOpenAI doesn’t say. It only visits pages submitted as ads, and isn’t used for training. | Overview of OpenAI Crawlersread October 9, 2026 |
CCBotCommon Crawl | Builds Common Crawl’s open repository of web crawl data, which anyone can access and analyze | YesCommon Crawl documents a robots.txt disallow as how to stop it crawling your site. | CCBotread October 9, 2026 |
The growth engine for the agentic web.
Get understood. Get cited. Get chosen.
Text Trovance
Or, in iMessage, send this note to +1 (839) 205-9508:
Hi Trovance, show me how AI sees my company.
You choose to send it. Trovance replies, and sends the updates you ask for, over iMessage through our messaging provider. Reply STOP anytime. Tapping or sending never bills you.