# Botcard > Cited robots.txt tokens for AI crawlers. One card per token, each fact tied to the vendor page it came from. Not legal advice. Botcard does not fetch third-party sites. Dataset 2026-10-18.9. Version 2026-10-18.9 adds the remaining Yes rows on Yandex's robot page, YandexPagechecker and YandexImageResizer, and the plain No rows: YandexAccessibilityBot, YandexCalendar, YandexCheckBot, YandexDirect, YandexDirectDyn, YandexFavicons, YandexFeed, YandexMobileBot, YandexMobileScreenShotBot, YandexOntoDBAPI, YandexPartner, YandexRCA, YandexSearchShop, YandexTracker, YandexUserproxy, and YandexScreenshotBot. A plain No means the robot does not take general User-agent: * rules into account. YandexCalendar is the page's example of a named directive. It is not every robot on that page. YandexMarket is not a card: the page lists YandexMarket/1.0 as taking general rules into account and YandexMarket/2.0 as not. YaDirectFetcher, the YandexMetrika/2.0 string that includes yabs01, and YandexMetrika/4.0 are not cards: the page says those robots do not use robots.txt. The other YandexMetrika/2.0 string and YandexMetrika/3.0 are a plain No. Those rows share the YandexMetrika token with the rows that do not use robots.txt, so there is no YandexMetrika card. YandexRenderResourcesBot is not a card: the page says it ignores robots.txt for a resource when the HTML page is allowed to the Yandex bot, and does not fetch that resource when the HTML is disallowed. This check does not apply that rule. Brave's crawler help says the crawler does not advertise a differentiated user agent, so there is no Brave card. It is not a full list of every Google, Apple, Amazon, Meta, Perplexity, DuckDuckGo, Common Crawl, ByteDance, Mistral, Diffbot, Ai2, Hive, or Mojeek crawler. Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google, Google-InspectionTool, GoogleOther, GoogleOther-Image, and GoogleOther-Video are on Google's common-crawlers page and are not cards. Google-NotebookLM is the former Gemini Notebook agent, supported until August 2026, and is not a card. FacebookExternalHit, Meta-ExternalAds, and Apple's iTMS string are not cards. Cohere's page lists no current bot; the Coherebot example is not a card. https://you.com/docs/youbot returned HTTP 403, so there is no YouBot card. ## API - [Crawler registry](https://whosthatbot.com/v1/crawlers): JSON array of every card - [Robots.txt check](https://whosthatbot.com/v1/robots-check): POST robots_txt and path. Does not fetch the host - [MCP endpoint](https://whosthatbot.com/mcp): POST JSON-RPC tools/call with name check_robots. Copy: `curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["gptbot"]}}}'`. Also list_crawlers and get_crawler. Does not fetch the host. No SSE stream - [MCP server card](https://whosthatbot.com/mcp/server-card): discovery card for the streamable HTTP endpoint - [Checker page](https://whosthatbot.com/check.html): the same check in a browser - [GPTBot](https://whosthatbot.com/v1/crawlers/gptbot): training - [OAI-SearchBot](https://whosthatbot.com/v1/crawlers/oai-searchbot): search - [OAI-AdsBot](https://whosthatbot.com/v1/crawlers/oai-adsbot): ads_check - [ChatGPT-User](https://whosthatbot.com/v1/crawlers/chatgpt-user): user_fetch - [Googlebot](https://whosthatbot.com/v1/crawlers/googlebot): search - [Google-Extended](https://whosthatbot.com/v1/crawlers/google-extended): control_token - [Google-CloudVertexBot](https://whosthatbot.com/v1/crawlers/google-cloudvertexbot): owner_requested - [ClaudeBot](https://whosthatbot.com/v1/crawlers/claudebot): training - [Claude-User](https://whosthatbot.com/v1/crawlers/claude-user): user_fetch - [Claude-SearchBot](https://whosthatbot.com/v1/crawlers/claude-searchbot): search - [Applebot](https://whosthatbot.com/v1/crawlers/applebot): search - [Applebot-Extended](https://whosthatbot.com/v1/crawlers/applebot-extended): control_token - [Perplexity-User](https://whosthatbot.com/v1/crawlers/perplexity-user): user_fetch - [PerplexityBot](https://whosthatbot.com/v1/crawlers/perplexitybot): search - [CCBot](https://whosthatbot.com/v1/crawlers/ccbot): open_crawl - [Amazonbot](https://whosthatbot.com/v1/crawlers/amazonbot): training - [Amzn-SearchBot](https://whosthatbot.com/v1/crawlers/amzn-searchbot): search - [Amzn-User](https://whosthatbot.com/v1/crawlers/amzn-user): user_fetch - [Meta-ExternalAgent](https://whosthatbot.com/v1/crawlers/meta-externalagent): training - [Meta-ExternalFetcher](https://whosthatbot.com/v1/crawlers/meta-externalfetcher): user_fetch - [Meta-WebIndexer](https://whosthatbot.com/v1/crawlers/meta-webindexer): search - [DuckAssistBot](https://whosthatbot.com/v1/crawlers/duckassistbot): search - [DuckDuckBot](https://whosthatbot.com/v1/crawlers/duckduckbot): search - [Bytespider](https://whosthatbot.com/v1/crawlers/bytespider): search - [MistralAI-User](https://whosthatbot.com/v1/crawlers/mistralai-user): user_fetch - [MistralAI-Index](https://whosthatbot.com/v1/crawlers/mistralai-index): search - [MistralAI-Training](https://whosthatbot.com/v1/crawlers/mistralai-training): training - [Diffbot](https://whosthatbot.com/v1/crawlers/diffbot): search - [Diffbot-User](https://whosthatbot.com/v1/crawlers/diffbot-user): user_fetch - [AI2Bot](https://whosthatbot.com/v1/crawlers/ai2bot): training - [ImagesiftBot](https://whosthatbot.com/v1/crawlers/imagesiftbot): search - [MojeekBot](https://whosthatbot.com/v1/crawlers/mojeekbot): search - [Google-GeminiNotebook](https://whosthatbot.com/v1/crawlers/google-gemininotebook): user_fetch - [Google-Agent](https://whosthatbot.com/v1/crawlers/google-agent): user_fetch - [YandexBot](https://whosthatbot.com/v1/crawlers/yandexbot): search - [YandexImages](https://whosthatbot.com/v1/crawlers/yandeximages): search - [YandexVideo](https://whosthatbot.com/v1/crawlers/yandexvideo): search - [YandexMedia](https://whosthatbot.com/v1/crawlers/yandexmedia): search - [YandexBlogs](https://whosthatbot.com/v1/crawlers/yandexblogs): search - [YandexVideoParser](https://whosthatbot.com/v1/crawlers/yandexvideoparser): search - [YandexComBot](https://whosthatbot.com/v1/crawlers/yandexcombot): search - [YandexAdditionalBot](https://whosthatbot.com/v1/crawlers/yandexadditionalbot): search - [YandexAdditional](https://whosthatbot.com/v1/crawlers/yandexadditional): search - [YandexVertis](https://whosthatbot.com/v1/crawlers/yandexvertis): search - [YandexVerticals](https://whosthatbot.com/v1/crawlers/yandexverticals): search - [YandexOntoDB](https://whosthatbot.com/v1/crawlers/yandexontodb): search - [YandexSitelinks](https://whosthatbot.com/v1/crawlers/yandexsitelinks): search - [YandexSpravBot](https://whosthatbot.com/v1/crawlers/yandexspravbot): search - [YandexWebmaster](https://whosthatbot.com/v1/crawlers/yandexwebmaster): search - [YandexAdNet](https://whosthatbot.com/v1/crawlers/yandexadnet): ads_check - [YandexAccessibilityBot](https://whosthatbot.com/v1/crawlers/yandexaccessibilitybot): search - [YandexCalendar](https://whosthatbot.com/v1/crawlers/yandexcalendar): user_fetch - [YandexCheckBot](https://whosthatbot.com/v1/crawlers/yandexcheckbot): search - [YandexDirect](https://whosthatbot.com/v1/crawlers/yandexdirect): ads_check - [YandexDirectDyn](https://whosthatbot.com/v1/crawlers/yandexdirectdyn): ads_check - [YandexFavicons](https://whosthatbot.com/v1/crawlers/yandexfavicons): search - [YandexFeed](https://whosthatbot.com/v1/crawlers/yandexfeed): owner_requested - [YandexImageResizer](https://whosthatbot.com/v1/crawlers/yandeximageresizer): search - [YandexMobileBot](https://whosthatbot.com/v1/crawlers/yandexmobilebot): search - [YandexMobileScreenShotBot](https://whosthatbot.com/v1/crawlers/yandexmobilescreenshotbot): search - [YandexOntoDBAPI](https://whosthatbot.com/v1/crawlers/yandexontodbapi): search - [YandexPagechecker](https://whosthatbot.com/v1/crawlers/yandexpagechecker): search - [YandexPartner](https://whosthatbot.com/v1/crawlers/yandexpartner): search - [YandexRCA](https://whosthatbot.com/v1/crawlers/yandexrca): search - [YandexSearchShop](https://whosthatbot.com/v1/crawlers/yandexsearchshop): user_fetch - [YandexTracker](https://whosthatbot.com/v1/crawlers/yandextracker): search - [YandexUserproxy](https://whosthatbot.com/v1/crawlers/yandexuserproxy): user_fetch - [YandexScreenshotBot](https://whosthatbot.com/v1/crawlers/yandexscreenshotbot): search ## Pages - [Overview](https://whosthatbot.com/): what the tokens mean, with the Google-Extended answer - [Overview markdown](https://whosthatbot.com/index.html.md) - [GPTBot page](https://whosthatbot.com/crawlers/gptbot.html): OpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot means the site's content should not be used for that training. OpenAI says this setting is independent of OAI-SearchBot. - [OAI-SearchBot page](https://whosthatbot.com/crawlers/oai-searchbot.html): OpenAI says OAI-SearchBot is used to surface sites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says this setting is independent of GPTBot, and that a robots.txt change can take about 24 hours to affect search. - [OAI-AdsBot page](https://whosthatbot.com/crawlers/oai-adsbot.html): OpenAI says OAI-AdsBot visits landing pages submitted as ChatGPT ads, to check policy compliance and relevance. It says those visits are not used to train generative AI foundation models. The same page does not say whether robots.txt controls OAI-AdsBot. - [ChatGPT-User page](https://whosthatbot.com/crawlers/chatgpt-user.html): OpenAI says ChatGPT-User fetches pages when a person uses ChatGPT, a Custom GPT, or a GPT Action. It is not an automatic crawl, and OpenAI says robots.txt rules may not apply. It is not the token for ChatGPT search opt-out; that token is OAI-SearchBot. - [Googlebot page](https://whosthatbot.com/crawlers/googlebot.html): Google says Googlebot is a common crawler, so it obeys robots.txt on automatic crawls. Preferences for this token affect Google Search, including Discover and Search features, plus products such as Google Images, video mode, and Google News. Google-Extended is a different token and does not change Search inclusion. - [Google-Extended page](https://whosthatbot.com/crawlers/google-extended.html): Google says publishers use the Google-Extended robots.txt token to manage whether content Google has crawled may train future Gemini models (Gemini Apps and Vertex AI API for Gemini) and whether it may ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says this token does not change inclusion or ranking in Google Search. - [Google-CloudVertexBot page](https://whosthatbot.com/crawlers/google-cloudvertexbot.html): Google says Google-CloudVertexBot performs crawls a site owner requested in order to build Vertex AI Agents. It does not affect Google Search. On this crawler, a robots.txt rule matches if it names Google-CloudVertexBot or Googlebot; Google says matching one of a crawler's tokens is enough. - [ClaudeBot page](https://whosthatbot.com/crawlers/claudebot.html): Anthropic says ClaudeBot collects public web content that may be used to train generative models. Restricting it signals that the site's future material should be left out of those training sets. The article says Anthropic's bots honor robots.txt, including a non-standard Crawl-delay. - [Claude-User page](https://whosthatbot.com/crawlers/claude-user.html): Anthropic says Claude-User fetches sites when a person asks Claude something. Disabling the token stops Anthropic from retrieving that content for the user query, which can reduce visibility in user-directed web search. This article treats the token as a robots.txt control. OpenAI's own ChatGPT-User page says robots.txt may not apply to that different token. - [Claude-SearchBot page](https://whosthatbot.com/crawlers/claude-searchbot.html): Anthropic says Claude-SearchBot crawls to improve search-result quality. Disabling it stops indexing for that search use and may reduce how visible or accurate the site is in search answers. It is a different token from ClaudeBot (training) and Claude-User (a person asked). - [Applebot page](https://whosthatbot.com/crawlers/applebot.html): Apple says Applebot crawls the web for search in Spotlight, Siri, and Safari. Enabling the token in robots.txt allows site content to appear in those results. The same page says crawled data may also train Apple foundation models, and that opting out of that training is the Applebot-Extended token, not this one. A nosnippet tag, not this token, is how the page says to opt out of broad world-knowledge answers. Apple says this crawler respects robots.txt on general search crawls aimed at Applebot, does not follow crawl-delay, and follows the Googlebot group when the file does not mention Applebot but does mention Googlebot. The page does not say that a Disallow removes the site from search. The iTMS user agent on the same page is not this token. Applebot-Extended is a different token and is not this card. - [Applebot-Extended page](https://whosthatbot.com/crawlers/applebot-extended.html): Apple says Applebot-Extended does not crawl pages. It only decides how Apple may use pages that Applebot already crawled: disallowing the token opts the site out of training Apple's general-purpose foundation models that power generative features in Apple Intelligence, Services, and Developer Tools. Apple says those pages can still be included in search results, and that this token is not a Search ranking signal. A nosnippet tag, not this token, is how the same page says to opt out of broad world-knowledge answers. - [Perplexity-User page](https://whosthatbot.com/crawlers/perplexity-user.html): Perplexity says Perplexity-User visits a page when a person asks a question, so it can answer and link that page. It says this token is not used to crawl the web or to collect content for training foundation models. The same section says the token controls which sites those requests can access, and that the fetcher generally ignores robots.txt because a person requested the fetch. Perplexity says each robots.txt setting works independently. PerplexityBot is a different token and is not this card. - [PerplexityBot page](https://whosthatbot.com/crawlers/perplexitybot.html): Perplexity says PerplexityBot surfaces and links sites in Perplexity search results. It says this token is not used to crawl content for foundation models. The same section recommends allowing the token in robots.txt, and allowing requests from the published IP ranges, so the site appears in those results. Perplexity says each robots.txt setting works independently and that a change can take up to 24 hours. The page's machine-readable article metadata lists dateModified 2026-01-29 and no datePublished. Perplexity-User is a different token and generally ignores robots.txt. This card does not. - [CCBot page](https://whosthatbot.com/crawlers/ccbot.html): Common Crawl's CCBot page says the crawler identifies itself as CCBot/2.0 with the FAQ URL inside the user-agent. The page description says it is a Nutch-based crawler that uses the crawl database to pick candidates. The body says the crawl supports an open repository of web crawl data, and that publishing User-agent: CCBot and Disallow: / prevents Common Crawl from crawling the site. The page says some crawlers falsely use the CCBot name, and that the real crawler uses dedicated IP ranges with reverse DNS, except IPv6 where reverse DNS is not yet supported. The CCBot page shows no article date. A separate post dated September 17, 2025 says the Opt-Out Registry is now the Opt-Out Ledger, because the list is immutable. That post says Common Crawl publishes every legal opt-out request it has received, encourages robots.txt to control crawling, and still processes legal requests. It points CCBot configuration back at the CCBot page and does not say the ledger replaces the Disallow block. - [Amazonbot page](https://whosthatbot.com/crawlers/amazonbot.html): Amazon says Amazonbot is used to improve products and services, to give customers more accurate information, and that it may be used to train Amazon AI models. The page says each of Amazonbot, Amzn-SearchBot, and Amzn-User is an independent setting, and that a change may take about 24 hours. Automated crawling by these user agents honors allow and disallow, does not support crawl-delay, and respects the noarchive, noindex, and none robots meta tags. Amzn-SearchBot and Amzn-User are different tokens. - [Amzn-SearchBot page](https://whosthatbot.com/crawlers/amzn-searchbot.html): Amazon says Amzn-SearchBot improves search in Amazon products and services. Permitting it makes a site eligible to appear in search experiences such as Alexa. Amazon says this token does not crawl content for generative AI model training. If robots.txt does not mention Amzn-SearchBot but allows other search bots, Amazon says this crawler follows those bots' directives. This check does not apply that fallback. Automated crawling honors allow and disallow and does not support crawl-delay. Each Amazon user-agent setting is independent. Amazonbot and Amzn-User are different tokens. - [Amzn-User page](https://whosthatbot.com/crawlers/amzn-user.html): Amazon says Amzn-User supports user actions, such as an Alexa query that needs current information, and may fetch a live page on that person's behalf. Because a person can start the action, Amazon says it may not follow all robots.txt directives. Amazon says this token does not crawl content for generative AI model training. The same page says automated crawling by the listed user agents honors allow and disallow. This card follows the Amzn-User paragraph. The checker still reports the RFC 9309 match and does not fetch a host. Amazonbot and Amzn-SearchBot are different tokens. - [Meta-ExternalAgent page](https://whosthatbot.com/crawlers/meta-externalagent.html): Meta says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. The page says to block a crawler by adding a Disallow for it, and that a robots.txt change can take up to 24 hours because crawlers may cache the file. The robots.txt example on the page uses User-agent: meta-externalagent. Meta-ExternalFetcher is a different token and may bypass robots.txt. Meta-WebIndexer is a different token. - [Meta-ExternalFetcher page](https://whosthatbot.com/crawlers/meta-externalfetcher.html): Meta says Meta-ExternalFetcher fetches individual links at a person's request. It supports product functions such as evaluating and improving agentic AI, including helping AI navigate a site to complete tasks. Meta says this crawler may bypass robots.txt because a person requested the fetch. The checker still reports the RFC 9309 match and does not fetch a host. Meta-ExternalAgent and Meta-WebIndexer are different tokens. - [Meta-WebIndexer page](https://whosthatbot.com/crawlers/meta-webindexer.html): Meta says Meta-WebIndexer crawls to improve Meta AI search result quality, and that allowing the token helps Meta cite and link the site in Meta AI responses. The page says to block a crawler by adding a Disallow, and that a robots.txt change can take up to 24 hours. Meta-ExternalAgent is the training token on the same page. Meta-ExternalFetcher may bypass robots.txt and is not this card. - [DuckAssistBot page](https://whosthatbot.com/crawlers/duckassistbot.html): DuckDuckGo says DuckAssistBot crawls pages in real time for AI-assisted answers that cite their sources, and that this data is not used to train AI models. A publisher opts out by disallowing the DuckAssistBot user agent. DuckDuckGo says that change takes effect after 72 hours and does not change organic search rankings or whether the site appears in search results. The page lists crawling@duckduckgo.com for opt-out problems. DuckDuckBot is a different token. - [DuckDuckBot page](https://whosthatbot.com/crawlers/duckduckbot.html): DuckDuckGo says DuckDuckBot improves search results. The page says the user agent will appear as DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) and that it respects WWW::RobotRules, linking that name to the MetaCPAN module page. DuckAssistBot is a different token: a real-time crawl for AI-assisted answers, not this card. - [Bytespider page](https://whosthatbot.com/crawlers/bytespider.html): Toutiao Search's webmaster page says Bytespider crawls pages, builds an index, and answers search queries. The page does not mention generative-model training or robots.txt. A separate Robots matching page (https://zhanzhang.toutiao.com/docs/intro/520) describes * and $ wildcards, path rules it says match Google's, and unordered query-parameter matching. That page does not name Bytespider and does not say the crawler honors robots.txt. The About page says a reverse-DNS hostname ending in .bytedance.com identifies the crawler, and that a crawl-frequency limit set in the webmaster tool takes effect within one day. Questions go to zhanzhang@bytedance.com. - [MistralAI-User page](https://whosthatbot.com/crawlers/mistralai-user.html): Mistral says MistralAI-User is for user actions in Vibe. When a person asks Vibe a question, it may visit a page and include a link to the source. Mistral says this token governs which sites those requests can be made to. It is not an automatic crawl and is not used to crawl content for generative AI training. MistralAI-Index and MistralAI-Training are different tokens. - [MistralAI-Index page](https://whosthatbot.com/crawlers/mistralai-index.html): Mistral says MistralAI-Index automatically crawls the web for indexing only, so Mistral search can help answer questions in Vibe. Mistral says that content is not used for generative AI training of any kind. The page lists this token as a robots.txt tag and does not say that the crawler obeys the file or that a Disallow stops indexing. MistralAI-User and MistralAI-Training are different tokens. - [MistralAI-Training page](https://whosthatbot.com/crawlers/mistralai-training.html): Mistral says MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models, and that webmasters can disallow this user agent in robots.txt. Mistral says this crawler is not used for search indexing or to answer live user queries in Vibe. MistralAI-User and MistralAI-Index are different tokens. - [Diffbot page](https://whosthatbot.com/crawlers/diffbot.html): Diffbot says its web crawls follow a site's robots.txt by default, including Disallow and crawl-delay, and that a partnership or agreement with the site can override that. This check does not apply that override and does not apply crawl-delay. Diffbot says this user agent is general proactive crawling so sites can be discovered and cited from the Diffbot Knowledge Graph and web search, and that it is not used for AI training. Diffbot-User is a different token. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card. - [Diffbot-User page](https://whosthatbot.com/crawlers/diffbot-user.html): Diffbot says Diffbot-User is for requests on behalf of a person browsing a URL in Diffbot software, in response to that person's input. The page does not say whether this token obeys robots.txt. The default adherence sentence on the same page is about Diffbot web crawls, which are the Diffbot token, not this one. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card. - [AI2Bot page](https://whosthatbot.com/crawlers/ai2bot.html): Ai2 says the AI2 Bot explores certain domains to find web content used to train open language models. The page publishes one HTTP user-agent and says that string can be used to filter or reject the crawler's traffic. It does not say whether robots.txt controls this token. - [ImagesiftBot page](https://whosthatbot.com/crawlers/imagesiftbot.html): The Imagesift by Hive page says ImageSiftBot scrapes publicly available images for its web intelligence products, and that those products use the saved index for search and retrieval of similar images. Standard robots.txt directives that target ImagesiftBot are respected. If there is no ImagesiftBot rule and there is a Googlebot rule, the page says ImagesiftBot follows the Googlebot directives. This check does not apply that fallback. The page says ImagesiftBot supports crawl-delay as a minimum number of seconds between the start of consecutive requests. This check does not apply crawl-delay. - [MojeekBot page](https://whosthatbot.com/crawlers/mojeekbot.html): Mojeek says MojeekBot is the web crawler for the Mojeek search engine. It obeys the Robot Exclusion Standard by taking the first record whose User-Agent contains MojeekBot, otherwise the first User-Agent * record, and it does not support the crawl-delay directive. This check matches the whole token and merges every group that names it, which is RFC 9309, and it does not apply crawl-delay. The page also says a noindex meta tag is fetched but not added to the search database. This check does not read meta tags. - [Google-GeminiNotebook page](https://whosthatbot.com/crawlers/google-gemininotebook.html): Google says the Gemini Notebook fetcher requests individual URLs that Gemini Notebook users have provided as sources for their projects. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page publishes mobile and desktop HTTP user-agent strings and does not print a separate robots.txt User-agent line. The same section lists Google-NotebookLM as a former agent supported until August 2026. This card does not include that string. - [Google-Agent page](https://whosthatbot.com/crawlers/google-agent.html): Google says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page publishes mobile and desktop HTTP user-agent strings and does not print a separate robots.txt User-agent line. That section says Google is also experimenting with Web Bot Auth, using the https://agent.bot.goog identity. This check does not verify that signature. - [YandexBot page](https://whosthatbot.com/crawlers/yandexbot.html): Yandex says YandexBot is the main indexing robot and that it takes the general rules in robots.txt into account. The MirrorDetector string is this token, not a separate card. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexImages page](https://whosthatbot.com/crawlers/yandeximages.html): Yandex says YandexImages indexes images to display them in Yandex Images and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexVideo page](https://whosthatbot.com/crawlers/yandexvideo.html): Yandex says YandexVideo indexes videos for Yandex video search and takes the general rules in robots.txt into account. YandexVideoParser is a different token and does not take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexMedia page](https://whosthatbot.com/crawlers/yandexmedia.html): Yandex says YandexMedia indexes multimedia data and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexBlogs page](https://whosthatbot.com/crawlers/yandexblogs.html): Yandex says YandexBlogs is the blog search robot that indexes post comments and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexVideoParser page](https://whosthatbot.com/crawlers/yandexvideoparser.html): Yandex says YandexVideoParser indexes videos for Yandex video search and does not take the general rules in robots.txt into account. YandexVideo is a different token and does take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexComBot page](https://whosthatbot.com/crawlers/yandexcombot.html): Yandex says YandexComBot indexes content for search in languages other than Russian. It can index that content if there is no explicit robot-specific restriction, and it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexAdditionalBot page](https://whosthatbot.com/crawlers/yandexadditionalbot.html): Yandex's Search with Yandex AI page says a Disallow for YandexAdditional or YandexAdditionalBot keeps that page out of Yandex AI answers. The example disallows one path under User-agent: YandexAdditionalBot, and the page says the change applies after a recrawl, typically 2–14 days. The robot list says these tokens help process robots.txt for that purpose, apply to pages the primary crawler has already indexed, do not make indexing requests, and do not take the general rules into account. https://yandex.com/support/webmaster/en/yandex-ai The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexAdditional page](https://whosthatbot.com/crawlers/yandexadditional.html): Yandex's Search with Yandex AI page says a Disallow for YandexAdditional or YandexAdditionalBot keeps that page out of Yandex AI answers. The example disallows one path under User-agent: YandexAdditionalBot, and the page says the change applies after a recrawl, typically 2–14 days. The robot list says these tokens help process robots.txt for that purpose, apply to pages the primary crawler has already indexed, do not make indexing requests, and do not take the general rules into account. https://yandex.com/support/webmaster/en/yandex-ai The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexVertis page](https://whosthatbot.com/crawlers/yandexvertis.html): Yandex says YandexVertis is the search verticals robot and that it takes the general rules in robots.txt into account. YandexVerticals is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexVerticals page](https://whosthatbot.com/crawlers/yandexverticals.html): Yandex says YandexVerticals is the robot of Yandex Verticals (Classifieds): Auto.ru, Yandex Realty, Yandex Jobs, and Yandex Reviews, and that it takes the general rules in robots.txt into account. YandexVertis is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexOntoDB page](https://whosthatbot.com/crawlers/yandexontodb.html): Yandex says YandexOntoDB is the information card robot and that it takes the general rules in robots.txt into account. YandexOntoDBAPI is a different token that downloads dynamic data and does not take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexSitelinks page](https://whosthatbot.com/crawlers/yandexsitelinks.html): Yandex says YandexSitelinks checks the availability of pages used as sitelinks and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexSpravBot page](https://whosthatbot.com/crawlers/yandexspravbot.html): Yandex says YandexSpravBot is the Yandex Business robot and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexWebmaster page](https://whosthatbot.com/crawlers/yandexwebmaster.html): Yandex says YandexWebmaster is the Yandex Webmaster robot and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexAdNet page](https://whosthatbot.com/crawlers/yandexadnet.html): Yandex says YandexAdNet is the Yandex Advertising Network robot and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexAccessibilityBot page](https://whosthatbot.com/crawlers/yandexaccessibilitybot.html): Yandex says YandexAccessibilityBot downloads pages to check their accessibility for users. It sends up to 3 requests to the site per second and ignores the crawl-rate setting in Yandex Webmaster. The robot list says it does not take the general rules in robots.txt into account. This check does not apply that crawl-rate setting. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexCalendar page](https://whosthatbot.com/crawlers/yandexcalendar.html): Yandex says YandexCalendar is the Yandex Calendar robot. It downloads calendar files by users' requests, and those files are often located in directories prohibited from indexing. The robot list says it does not take the general rules in robots.txt into account. The page's example is User-agent: YandexCalendar and Disallow: /. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexCheckBot page](https://whosthatbot.com/crawlers/yandexcheckbot.html): Yandex says YandexCheckBot downloads pages from websites in different regions of the world to check their availability to users and does not make indexing requests. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexDirect page](https://whosthatbot.com/crawlers/yandexdirect.html): Yandex says YandexDirect downloads information about the content of Yandex Advertising network partner sites to identify their topic categories to match relevant advertising. The robot list says it does not take the general rules in robots.txt into account. YaDirectFetcher is a different token: the page says that robot does not use robots.txt. YandexDirectDyn is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexDirectDyn page](https://whosthatbot.com/crawlers/yandexdirectdyn.html): Yandex says YandexDirectDyn generates dynamic banners. The robot list says it does not take the general rules in robots.txt into account. YandexDirect is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexFavicons page](https://whosthatbot.com/crawlers/yandexfavicons.html): Yandex says YandexFavicons downloads the site's favicon file to display in search results. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexFeed page](https://whosthatbot.com/crawlers/yandexfeed.html): Yandex says YandexFeed downloads structured data via links that website owners submitted to Yandex services and does not make indexing requests. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexImageResizer page](https://whosthatbot.com/crawlers/yandeximageresizer.html): Yandex says YandexImageResizer is the mobile devices robot and that it takes the general rules in robots.txt into account. YandexImages is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexMobileBot page](https://whosthatbot.com/crawlers/yandexmobilebot.html): Yandex says YandexMobileBot defines pages with layout suitable for mobile devices. The robot list says it does not take the general rules in robots.txt into account. The page's example is User-agent: YandexMobileBot and Disallow: /private/*.txt$. YandexMobileScreenShotBot is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexMobileScreenShotBot page](https://whosthatbot.com/crawlers/yandexmobilescreenshotbot.html): Yandex says YandexMobileScreenShotBot takes a screenshot of the mobile page. The robot list says it does not take the general rules in robots.txt into account. YandexMobileBot and YandexScreenshotBot are different tokens. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexOntoDBAPI page](https://whosthatbot.com/crawlers/yandexontodbapi.html): Yandex says YandexOntoDBAPI is the information card robot that downloads dynamic data. The robot list says it does not take the general rules in robots.txt into account. YandexOntoDB is a different token and does take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexPagechecker page](https://whosthatbot.com/crawlers/yandexpagechecker.html): Yandex says YandexPagechecker accesses the page for validating structured data via the Structured data validator and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexPartner page](https://whosthatbot.com/crawlers/yandexpartner.html): Yandex says YandexPartner downloads information about the content of Yandex partner sites. The robot list says it does not take the general rules in robots.txt into account. The page does not name an advertising product for this token. YandexDirect is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexRCA page](https://whosthatbot.com/crawlers/yandexrca.html): Yandex says YandexRCA collects data for generating previews. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexSearchShop page](https://whosthatbot.com/crawlers/yandexsearchshop.html): Yandex says YandexSearchShop downloads product catalogs in YML files by users' requests. These files are often placed in directories prohibited for indexing. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexTracker page](https://whosthatbot.com/crawlers/yandextracker.html): Yandex says YandexTracker is the Yandex Tracker robot. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexUserproxy page](https://whosthatbot.com/crawlers/yandexuserproxy.html): Yandex says YandexUserproxy proxies user actions on Yandex services. It sends requests in response to button clicks and downloads pages for online translation. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. - [YandexScreenshotBot page](https://whosthatbot.com/crawlers/yandexscreenshotbot.html): Yandex says YandexScreenshotBot takes a screenshot of the page. The robot list says it does not take the general rules in robots.txt into account. YandexMobileScreenShotBot is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement. ## Optional - [OpenAPI](https://whosthatbot.com/openapi.json): the registry GETs, the robots.txt check, and the MCP endpoint - [AI catalog](https://whosthatbot.com/.well-known/ai-catalog.json): points at the MCP server card - [Privacy](https://whosthatbot.com/privacy.html) - [Terms](https://whosthatbot.com/terms.html)