GPTBot
OpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot means the site's content should not be used for that training. OpenAI says this setting is independent of OAI-SearchBot.
Cited robots.txt tokens for AI crawlers, so an agent can tell training, search, a person-triggered fetch, and an open crawl apart.
Google-Extended is a robots.txt control for Gemini training and grounding. It has no HTTP user-agent of its own, and Google says it does not change whether a page stays in Google Search. The search crawler token is Googlebot. Card checked 2026-10-17 against Google's common crawlers list (page updated 2026-10-15).
Applebot is Apple's search crawler for Spotlight, Siri, and Safari. Apple says it respects robots.txt on general search crawls aimed at that token, does not follow crawl-delay, and follows a Googlebot group only when the file does not mention Applebot. The checker reports that fallback and does not apply it. Applebot-Extended is a different token: a control for foundation-model training that does not crawl, and disallowing it does not remove a page from Apple search. Applebot was checked 2026-10-18 and Applebot-Extended 2026-10-17 against About Applebot (page published 2026-09-04).
PerplexityBot surfaces sites in Perplexity search. Perplexity says it is not used to train foundation models, and recommends allowing the token in robots.txt so the site appears in search results. Perplexity-User is a different token: a person-triggered fetch that generally ignores robots.txt. Both cards were checked 2026-10-17 against Perplexity Crawlers (page metadata dateModified 2026-01-29).
CCBot is Common Crawl's crawler for an open web-crawl repository. The CCBot page says publishing User-agent: CCBot and Disallow: / prevents the crawl, and that some crawlers falsely use that name. A post dated 2025-09-17 says the Opt-Out Registry is now the Opt-Out Ledger: Common Crawl encourages robots.txt and also publishes legal opt-out requests. The checker reports that ledger and does not apply it. The CCBot page was checked 2026-10-18 and shows no article date. The opt-out post was read the same day.
Amazon's page lists three independent tokens, checked 2026-10-18. Amazonbot may train Amazon AI models, and automated crawls honor allow and disallow. Amzn-SearchBot is for search such as Alexa, does not train generative models, and follows other search bots only when the file does not mention Amzn-SearchBot. The checker does not apply that fallback. Amzn-User fetches for a person's request and may not follow every robots.txt directive. Meta-ExternalAgent trains foundation models or indexes content. Meta-WebIndexer is for Meta AI search. Meta-ExternalFetcher fetches at a person's request and may bypass robots.txt. DuckAssistBot crawls for DuckDuckGo AI-assisted answers and is not used to train models; a Disallow takes effect after 72 hours and does not change search rankings. DuckDuckBot is the search crawler and respects WWW::RobotRules.
curl -s https://whosthatbot.com/v1/crawlers/perplexitybotThe registry is /v1/crawlers. 34 cards, 15 vendors, no key. Botcard does not fetch the site you are asking about.
Paste the file and a path. The checker answers allow or disallow for each token under RFC 9309, and it repeats that token's vendor stance. It does not fetch a host.
curl -s https://whosthatbot.com/v1/robots-check -H 'content-type: application/json' -d '{"robots_txt":"User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs"}'tools/call with name check_robots is the same check at POST /mcp. POST tools/call check_robots to /mcp. Send Content-Type: application/json and Accept: application/json, text/event-stream. Tools: check_robots, list_crawlers, get_crawler. All three are read-only. The discovery card is /mcp/server-card.
The pasted file disallows GPTBot on /docs and allows *. The call does not fetch a host.
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["gptbot"]}}}'That call returns fetched_target false. This row is disallow via group GPTBot, rule Disallow: /.
GPTBotOpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot means the site's content should not be used for that training. OpenAI says this setting is independent of OAI-SearchBot.
OAI-SearchBotOpenAI says OAI-SearchBot is used to surface sites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says this setting is independent of GPTBot, and that a robots.txt change can take about 24 hours to affect search.
OAI-AdsBotOpenAI says OAI-AdsBot visits landing pages submitted as ChatGPT ads, to check policy compliance and relevance. It says those visits are not used to train generative AI foundation models. The same page does not say whether robots.txt controls OAI-AdsBot.
ChatGPT-UserOpenAI says ChatGPT-User fetches pages when a person uses ChatGPT, a Custom GPT, or a GPT Action. It is not an automatic crawl, and OpenAI says robots.txt rules may not apply. It is not the token for ChatGPT search opt-out; that token is OAI-SearchBot.
GooglebotGoogle says Googlebot is a common crawler, so it obeys robots.txt on automatic crawls. Preferences for this token affect Google Search, including Discover and Search features, plus products such as Google Images, video mode, and Google News. Google-Extended is a different token and does not change Search inclusion.
Google-ExtendedGoogle says publishers use the Google-Extended robots.txt token to manage whether content Google has crawled may train future Gemini models (Gemini Apps and Vertex AI API for Gemini) and whether it may ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says this token does not change inclusion or ranking in Google Search.
Google-CloudVertexBotGoogle says Google-CloudVertexBot performs crawls a site owner requested in order to build Vertex AI Agents. It does not affect Google Search. On this crawler, a robots.txt rule matches if it names Google-CloudVertexBot or Googlebot; Google says matching one of a crawler's tokens is enough.
ClaudeBotAnthropic says ClaudeBot collects public web content that may be used to train generative models. Restricting it signals that the site's future material should be left out of those training sets. The article says Anthropic's bots honor robots.txt, including a non-standard Crawl-delay.
Claude-UserAnthropic says Claude-User fetches sites when a person asks Claude something. Disabling the token stops Anthropic from retrieving that content for the user query, which can reduce visibility in user-directed web search. This article treats the token as a robots.txt control. OpenAI's own ChatGPT-User page says robots.txt may not apply to that different token.
Claude-SearchBotAnthropic says Claude-SearchBot crawls to improve search-result quality. Disabling it stops indexing for that search use and may reduce how visible or accurate the site is in search answers. It is a different token from ClaudeBot (training) and Claude-User (a person asked).
ApplebotApple says Applebot crawls the web for search in Spotlight, Siri, and Safari. Enabling the token in robots.txt allows site content to appear in those results. The same page says crawled data may also train Apple foundation models, and that opting out of that training is the Applebot-Extended token, not this one. A nosnippet tag, not this token, is how the page says to opt out of broad world-knowledge answers. Apple says this crawler respects robots.txt on general search crawls aimed at Applebot, does not follow crawl-delay, and follows the Googlebot group when the file does not mention Applebot but does mention Googlebot. The page does not say that a Disallow removes the site from search. The iTMS user agent on the same page is not this token. Applebot-Extended is a different token and is not this card.
Applebot-ExtendedApple says Applebot-Extended does not crawl pages. It only decides how Apple may use pages that Applebot already crawled: disallowing the token opts the site out of training Apple's general-purpose foundation models that power generative features in Apple Intelligence, Services, and Developer Tools. Apple says those pages can still be included in search results, and that this token is not a Search ranking signal. A nosnippet tag, not this token, is how the same page says to opt out of broad world-knowledge answers.
Perplexity-UserPerplexity says Perplexity-User visits a page when a person asks a question, so it can answer and link that page. It says this token is not used to crawl the web or to collect content for training foundation models. The same section says the token controls which sites those requests can access, and that the fetcher generally ignores robots.txt because a person requested the fetch. Perplexity says each robots.txt setting works independently. PerplexityBot is a different token and is not this card.
PerplexityBotPerplexity says PerplexityBot surfaces and links sites in Perplexity search results. It says this token is not used to crawl content for foundation models. The same section recommends allowing the token in robots.txt, and allowing requests from the published IP ranges, so the site appears in those results. Perplexity says each robots.txt setting works independently and that a change can take up to 24 hours. The page's machine-readable article metadata lists dateModified 2026-01-29 and no datePublished. Perplexity-User is a different token and generally ignores robots.txt. This card does not.
CCBotCommon Crawl's CCBot page says the crawler identifies itself as CCBot/2.0 with the FAQ URL inside the user-agent. The page description says it is a Nutch-based crawler that uses the crawl database to pick candidates. The body says the crawl supports an open repository of web crawl data, and that publishing User-agent: CCBot and Disallow: / prevents Common Crawl from crawling the site. The page says some crawlers falsely use the CCBot name, and that the real crawler uses dedicated IP ranges with reverse DNS, except IPv6 where reverse DNS is not yet supported. The CCBot page shows no article date. A separate post dated September 17, 2025 says the Opt-Out Registry is now the Opt-Out Ledger, because the list is immutable. That post says Common Crawl publishes every legal opt-out request it has received, encourages robots.txt to control crawling, and still processes legal requests. It points CCBot configuration back at the CCBot page and does not say the ledger replaces the Disallow block.
AmazonbotAmazon says Amazonbot is used to improve products and services, to give customers more accurate information, and that it may be used to train Amazon AI models. The page says each of Amazonbot, Amzn-SearchBot, and Amzn-User is an independent setting, and that a change may take about 24 hours. Automated crawling by these user agents honors allow and disallow, does not support crawl-delay, and respects the noarchive, noindex, and none robots meta tags. Amzn-SearchBot and Amzn-User are different tokens.
Amzn-SearchBotAmazon says Amzn-SearchBot improves search in Amazon products and services. Permitting it makes a site eligible to appear in search experiences such as Alexa. Amazon says this token does not crawl content for generative AI model training. If robots.txt does not mention Amzn-SearchBot but allows other search bots, Amazon says this crawler follows those bots' directives. This check does not apply that fallback. Automated crawling honors allow and disallow and does not support crawl-delay. Each Amazon user-agent setting is independent. Amazonbot and Amzn-User are different tokens.
Amzn-UserAmazon says Amzn-User supports user actions, such as an Alexa query that needs current information, and may fetch a live page on that person's behalf. Because a person can start the action, Amazon says it may not follow all robots.txt directives. Amazon says this token does not crawl content for generative AI model training. The same page says automated crawling by the listed user agents honors allow and disallow. This card follows the Amzn-User paragraph. The checker still reports the RFC 9309 match and does not fetch a host. Amazonbot and Amzn-SearchBot are different tokens.
Meta-ExternalAgentMeta says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. The page says to block a crawler by adding a Disallow for it, and that a robots.txt change can take up to 24 hours because crawlers may cache the file. The robots.txt example on the page uses User-agent: meta-externalagent. Meta-ExternalFetcher is a different token and may bypass robots.txt. Meta-WebIndexer is a different token.
Meta-ExternalFetcherMeta says Meta-ExternalFetcher fetches individual links at a person's request. It supports product functions such as evaluating and improving agentic AI, including helping AI navigate a site to complete tasks. Meta says this crawler may bypass robots.txt because a person requested the fetch. The checker still reports the RFC 9309 match and does not fetch a host. Meta-ExternalAgent and Meta-WebIndexer are different tokens.
Meta-WebIndexerMeta says Meta-WebIndexer crawls to improve Meta AI search result quality, and that allowing the token helps Meta cite and link the site in Meta AI responses. The page says to block a crawler by adding a Disallow, and that a robots.txt change can take up to 24 hours. Meta-ExternalAgent is the training token on the same page. Meta-ExternalFetcher may bypass robots.txt and is not this card.
DuckAssistBotDuckDuckGo says DuckAssistBot crawls pages in real time for AI-assisted answers that cite their sources, and that this data is not used to train AI models. A publisher opts out by disallowing the DuckAssistBot user agent. DuckDuckGo says that change takes effect after 72 hours and does not change organic search rankings or whether the site appears in search results. The page lists crawling@duckduckgo.com for opt-out problems. DuckDuckBot is a different token.
DuckDuckBotDuckDuckGo says DuckDuckBot improves search results. The page says the user agent will appear as DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) and that it respects WWW::RobotRules, linking that name to the MetaCPAN module page. DuckAssistBot is a different token: a real-time crawl for AI-assisted answers, not this card.
BytespiderToutiao Search's webmaster page says Bytespider crawls pages, builds an index, and answers search queries. The page does not mention generative-model training or robots.txt. A separate Robots matching page (https://zhanzhang.toutiao.com/docs/intro/520) describes * and $ wildcards, path rules it says match Google's, and unordered query-parameter matching. That page does not name Bytespider and does not say the crawler honors robots.txt. The About page says a reverse-DNS hostname ending in .bytedance.com identifies the crawler, and that a crawl-frequency limit set in the webmaster tool takes effect within one day. Questions go to zhanzhang@bytedance.com.
MistralAI-UserMistral says MistralAI-User is for user actions in Vibe. When a person asks Vibe a question, it may visit a page and include a link to the source. Mistral says this token governs which sites those requests can be made to. It is not an automatic crawl and is not used to crawl content for generative AI training. MistralAI-Index and MistralAI-Training are different tokens.
MistralAI-IndexMistral says MistralAI-Index automatically crawls the web for indexing only, so Mistral search can help answer questions in Vibe. Mistral says that content is not used for generative AI training of any kind. The page lists this token as a robots.txt tag and does not say that the crawler obeys the file or that a Disallow stops indexing. MistralAI-User and MistralAI-Training are different tokens.
MistralAI-TrainingMistral says MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models, and that webmasters can disallow this user agent in robots.txt. Mistral says this crawler is not used for search indexing or to answer live user queries in Vibe. MistralAI-User and MistralAI-Index are different tokens.
DiffbotDiffbot says its web crawls follow a site's robots.txt by default, including Disallow and crawl-delay, and that a partnership or agreement with the site can override that. This check does not apply that override and does not apply crawl-delay. Diffbot says this user agent is general proactive crawling so sites can be discovered and cited from the Diffbot Knowledge Graph and web search, and that it is not used for AI training. Diffbot-User is a different token. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.
Diffbot-UserDiffbot says Diffbot-User is for requests on behalf of a person browsing a URL in Diffbot software, in response to that person's input. The page does not say whether this token obeys robots.txt. The default adherence sentence on the same page is about Diffbot web crawls, which are the Diffbot token, not this one. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.
AI2BotAi2 says the AI2 Bot explores certain domains to find web content used to train open language models. The page publishes one HTTP user-agent and says that string can be used to filter or reject the crawler's traffic. It does not say whether robots.txt controls this token.
ImagesiftBotThe Imagesift by Hive page says ImageSiftBot scrapes publicly available images for its web intelligence products, and that those products use the saved index for search and retrieval of similar images. Standard robots.txt directives that target ImagesiftBot are respected. If there is no ImagesiftBot rule and there is a Googlebot rule, the page says ImagesiftBot follows the Googlebot directives. This check does not apply that fallback. The page says ImagesiftBot supports crawl-delay as a minimum number of seconds between the start of consecutive requests. This check does not apply crawl-delay.
MojeekBotMojeek says MojeekBot is the web crawler for the Mojeek search engine. It obeys the Robot Exclusion Standard by taking the first record whose User-Agent contains MojeekBot, otherwise the first User-Agent * record, and it does not support the crawl-delay directive. This check matches the whole token and merges every group that names it, which is RFC 9309, and it does not apply crawl-delay. The page also says a noindex meta tag is fetched but not added to the search database. This check does not read meta tags.
Google-GeminiNotebookGoogle says the Gemini Notebook fetcher requests individual URLs that Gemini Notebook users have provided as sources for their projects. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page publishes mobile and desktop HTTP user-agent strings and does not print a separate robots.txt User-agent line. The same section lists Google-NotebookLM as a former agent supported until August 2026. This card does not include that string.
Google-AgentGoogle says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page publishes mobile and desktop HTTP user-agent strings and does not print a separate robots.txt User-agent line. That section says Google is also experimenting with Web Bot Auth, using the https://agent.bot.goog identity. This check does not verify that signature.