`Googlebot`

Google says Googlebot is a common crawler, so it obeys robots.txt on automatic crawls. Preferences for this token affect Google Search, including Discover and Search features, plus products such as Google Images, video mode, and Google News. Google-Extended is a different token and does not change Search inclusion.

robots.txt token: Googlebot
Operator: Google
Kind: search
robots.txt: obeys. The vendor says this crawler obeys robots.txt on automatic crawls.
HTTP user-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Google lists this short string as a rare form. The usual desktop and smartphone strings also contain Googlebot/2.1 and a Chrome version that changes. Match the token, not one full string.
IP list: https://developers.google.com/static/crawling/ipranges/common-crawlers.json Google says its common crawlers generally crawl from this list, and that a user-agent string can be spoofed. Verify with reverse DNS before trusting an IP. URL returned HTTP 200 on 2026-10-17. Botcard does not copy the list.
Source: List of Google's common crawlers. Retrieved 2026-10-17, page says updated 2026-10-15. CC BY 4.0 (Google Developers page, except as that page notes).

JSON

curl -s https://whosthatbot.com/v1/crawlers/googlebot

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Googlebot on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Googlebot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["googlebot"]}}}'

That call returns fetched_target false. This row is disallow via group Googlebot, rule Disallow: /.