# Botcard Cited robots.txt tokens for AI crawlers. Training, search, person-triggered fetches, and an open crawl are different tokens. Google-Extended is a robots.txt control for Gemini training and grounding. It has no HTTP user-agent of its own. Google says it does not change Google Search inclusion. See Googlebot for the search crawler. Source: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers (retrieved 2026-10-17, page updated 2026-10-15). Applebot is Apple's search crawler for Spotlight, Siri, and Safari. It respects robots.txt on general search crawls aimed at that token, does not follow crawl-delay, and follows a Googlebot group only when the file does not mention Applebot. The checker reports that fallback and does not apply it. Applebot-Extended is a different token and does not crawl. Source: https://support.apple.com/en-us/119829 (Applebot retrieved 2026-10-18, Applebot-Extended retrieved 2026-10-17, page published 2026-09-04). PerplexityBot surfaces sites in Perplexity search. Perplexity recommends allowing the token in robots.txt so the site appears in search results, and says the token is not used to train foundation models. Perplexity-User is a different token and generally ignores robots.txt. Source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers (retrieved 2026-10-17, page metadata dateModified 2026-01-29). CCBot is Common Crawl's crawler for an open web-crawl repository. The CCBot page says User-agent: CCBot and Disallow: / prevents the crawl, and that some crawlers falsely use that name. A post dated 2025-09-17 says the Opt-Out Registry is now the Opt-Out Ledger: Common Crawl encourages robots.txt and also publishes legal opt-out requests. The checker reports that ledger and does not apply it. Sources: https://commoncrawl.org/ccbot (retrieved 2026-10-18; the page shows no article date) and https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry (header date 2025-09-17, retrieved 2026-10-18). Amazon's page lists three independent tokens. Amazonbot may train Amazon AI models, and automated crawls honor allow and disallow. Amzn-SearchBot is for search such as Alexa, does not train generative models, and follows other search bots only when the file does not mention Amzn-SearchBot. The checker does not apply that fallback. Amzn-User fetches for a person's request and may not follow every robots.txt directive. Source: https://developer.amazon.com/amazonbot (retrieved 2026-10-18; the page shows no article date). Meta-ExternalAgent trains foundation models or indexes content. Meta-WebIndexer is for Meta AI search. Meta-ExternalFetcher fetches at a person's request and may bypass robots.txt. Source: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers (retrieved 2026-10-18; the page shows no article date). FacebookExternalHit and Meta-ExternalAds are on that page and are not cards. DuckAssistBot crawls for DuckDuckGo AI-assisted answers and is not used to train models. A Disallow takes effect after 72 hours and does not change search rankings. DuckDuckBot is the search crawler and respects WWW::RobotRules. Sources: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot and https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot (retrieved 2026-10-18; neither page shows an article date). Registry JSON: https://whosthatbot.com/v1/crawlers Robots.txt check (paste the file, no fetch): https://whosthatbot.com/v1/robots-check MCP streamable HTTP: https://whosthatbot.com/mcp MCP server card: https://whosthatbot.com/mcp/server-card Checker page: https://whosthatbot.com/check.html ## Check GPTBot over MCP tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows GPTBot on `/docs` and allows `*`. The call does not fetch a host. Other tools on the same endpoint: `list_crawlers`, `get_crawler`. ``` curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["gptbot"]}}}' ``` That call returns fetched_target false. This row is disallow via group GPTBot, rule Disallow: /. - [GPTBot](https://whosthatbot.com/crawlers/gptbot.html): training, robots.txt disallow_is_opt_out - [OAI-SearchBot](https://whosthatbot.com/crawlers/oai-searchbot.html): search, robots.txt disallow_is_opt_out - [OAI-AdsBot](https://whosthatbot.com/crawlers/oai-adsbot.html): ads_check, robots.txt not_stated - [ChatGPT-User](https://whosthatbot.com/crawlers/chatgpt-user.html): user_fetch, robots.txt may_not_apply - [Googlebot](https://whosthatbot.com/crawlers/googlebot.html): search, robots.txt obeys - [Google-Extended](https://whosthatbot.com/crawlers/google-extended.html): control_token, robots.txt control_token - [Google-CloudVertexBot](https://whosthatbot.com/crawlers/google-cloudvertexbot.html): owner_requested, robots.txt obeys - [ClaudeBot](https://whosthatbot.com/crawlers/claudebot.html): training, robots.txt disallow_is_opt_out - [Claude-User](https://whosthatbot.com/crawlers/claude-user.html): user_fetch, robots.txt disallow_is_opt_out - [Claude-SearchBot](https://whosthatbot.com/crawlers/claude-searchbot.html): search, robots.txt disallow_is_opt_out - [Applebot](https://whosthatbot.com/crawlers/applebot.html): search, robots.txt respects_no_crawl_delay - [Applebot-Extended](https://whosthatbot.com/crawlers/applebot-extended.html): control_token, robots.txt control_token - [Perplexity-User](https://whosthatbot.com/crawlers/perplexity-user.html): user_fetch, robots.txt generally_ignores - [PerplexityBot](https://whosthatbot.com/crawlers/perplexitybot.html): search, robots.txt recommends_allow - [CCBot](https://whosthatbot.com/crawlers/ccbot.html): open_crawl, robots.txt disallow_is_opt_out - [Amazonbot](https://whosthatbot.com/crawlers/amazonbot.html): training, robots.txt obeys - [Amzn-SearchBot](https://whosthatbot.com/crawlers/amzn-searchbot.html): search, robots.txt obeys - [Amzn-User](https://whosthatbot.com/crawlers/amzn-user.html): user_fetch, robots.txt may_not_apply - [Meta-ExternalAgent](https://whosthatbot.com/crawlers/meta-externalagent.html): training, robots.txt disallow_is_opt_out - [Meta-ExternalFetcher](https://whosthatbot.com/crawlers/meta-externalfetcher.html): user_fetch, robots.txt may_not_apply - [Meta-WebIndexer](https://whosthatbot.com/crawlers/meta-webindexer.html): search, robots.txt disallow_is_opt_out - [DuckAssistBot](https://whosthatbot.com/crawlers/duckassistbot.html): search, robots.txt disallow_is_opt_out - [DuckDuckBot](https://whosthatbot.com/crawlers/duckduckbot.html): search, robots.txt obeys - [Bytespider](https://whosthatbot.com/crawlers/bytespider.html): search, robots.txt not_stated - [MistralAI-User](https://whosthatbot.com/crawlers/mistralai-user.html): user_fetch, robots.txt governs_sites - [MistralAI-Index](https://whosthatbot.com/crawlers/mistralai-index.html): search, robots.txt published_tag - [MistralAI-Training](https://whosthatbot.com/crawlers/mistralai-training.html): training, robots.txt disallow_is_opt_out - [Diffbot](https://whosthatbot.com/crawlers/diffbot.html): search, robots.txt obeys_unless_agreed - [Diffbot-User](https://whosthatbot.com/crawlers/diffbot-user.html): user_fetch, robots.txt not_stated - [AI2Bot](https://whosthatbot.com/crawlers/ai2bot.html): training, robots.txt not_stated - [ImagesiftBot](https://whosthatbot.com/crawlers/imagesiftbot.html): search, robots.txt obeys - [MojeekBot](https://whosthatbot.com/crawlers/mojeekbot.html): search, robots.txt respects_no_crawl_delay - [Google-GeminiNotebook](https://whosthatbot.com/crawlers/google-gemininotebook.html): user_fetch, robots.txt generally_ignores - [Google-Agent](https://whosthatbot.com/crawlers/google-agent.html): user_fetch, robots.txt generally_ignores - [YandexBot](https://whosthatbot.com/crawlers/yandexbot.html): search, robots.txt obeys - [YandexImages](https://whosthatbot.com/crawlers/yandeximages.html): search, robots.txt obeys - [YandexVideo](https://whosthatbot.com/crawlers/yandexvideo.html): search, robots.txt obeys - [YandexMedia](https://whosthatbot.com/crawlers/yandexmedia.html): search, robots.txt obeys - [YandexBlogs](https://whosthatbot.com/crawlers/yandexblogs.html): search, robots.txt obeys - [YandexVideoParser](https://whosthatbot.com/crawlers/yandexvideoparser.html): search, robots.txt ignores_star - [YandexComBot](https://whosthatbot.com/crawlers/yandexcombot.html): search, robots.txt indexes_unless_named - [YandexAdditionalBot](https://whosthatbot.com/crawlers/yandexadditionalbot.html): search, robots.txt disallow_opts_out_ignores_star - [YandexAdditional](https://whosthatbot.com/crawlers/yandexadditional.html): search, robots.txt disallow_opts_out_ignores_star - [YandexVertis](https://whosthatbot.com/crawlers/yandexvertis.html): search, robots.txt obeys - [YandexVerticals](https://whosthatbot.com/crawlers/yandexverticals.html): search, robots.txt obeys - [YandexOntoDB](https://whosthatbot.com/crawlers/yandexontodb.html): search, robots.txt obeys - [YandexSitelinks](https://whosthatbot.com/crawlers/yandexsitelinks.html): search, robots.txt obeys - [YandexSpravBot](https://whosthatbot.com/crawlers/yandexspravbot.html): search, robots.txt obeys - [YandexWebmaster](https://whosthatbot.com/crawlers/yandexwebmaster.html): search, robots.txt obeys - [YandexAdNet](https://whosthatbot.com/crawlers/yandexadnet.html): ads_check, robots.txt obeys - [YandexAccessibilityBot](https://whosthatbot.com/crawlers/yandexaccessibilitybot.html): search, robots.txt ignores_star - [YandexCalendar](https://whosthatbot.com/crawlers/yandexcalendar.html): user_fetch, robots.txt ignores_star - [YandexCheckBot](https://whosthatbot.com/crawlers/yandexcheckbot.html): search, robots.txt ignores_star - [YandexDirect](https://whosthatbot.com/crawlers/yandexdirect.html): ads_check, robots.txt ignores_star - [YandexDirectDyn](https://whosthatbot.com/crawlers/yandexdirectdyn.html): ads_check, robots.txt ignores_star - [YandexFavicons](https://whosthatbot.com/crawlers/yandexfavicons.html): search, robots.txt ignores_star - [YandexFeed](https://whosthatbot.com/crawlers/yandexfeed.html): owner_requested, robots.txt ignores_star - [YandexImageResizer](https://whosthatbot.com/crawlers/yandeximageresizer.html): search, robots.txt obeys - [YandexMobileBot](https://whosthatbot.com/crawlers/yandexmobilebot.html): search, robots.txt ignores_star - [YandexMobileScreenShotBot](https://whosthatbot.com/crawlers/yandexmobilescreenshotbot.html): search, robots.txt ignores_star - [YandexOntoDBAPI](https://whosthatbot.com/crawlers/yandexontodbapi.html): search, robots.txt ignores_star - [YandexPagechecker](https://whosthatbot.com/crawlers/yandexpagechecker.html): search, robots.txt obeys - [YandexPartner](https://whosthatbot.com/crawlers/yandexpartner.html): search, robots.txt ignores_star - [YandexRCA](https://whosthatbot.com/crawlers/yandexrca.html): search, robots.txt ignores_star - [YandexSearchShop](https://whosthatbot.com/crawlers/yandexsearchshop.html): user_fetch, robots.txt ignores_star - [YandexTracker](https://whosthatbot.com/crawlers/yandextracker.html): search, robots.txt ignores_star - [YandexUserproxy](https://whosthatbot.com/crawlers/yandexuserproxy.html): user_fetch, robots.txt ignores_star - [YandexScreenshotBot](https://whosthatbot.com/crawlers/yandexscreenshotbot.html): search, robots.txt ignores_star Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.