# Googlebot

Google says Googlebot is a common crawler, so it obeys robots.txt on automatic crawls. Preferences for this token affect Google Search, including Discover and Search features, plus products such as Google Images, video mode, and Google News. Google-Extended is a different token and does not change Search inclusion.

- Operator: Google
- Kind: search
- robots.txt: obeys. The vendor says this crawler obeys robots.txt on automatic crawls.
- HTTP user-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Google lists this short string as a rare form. The usual desktop and smartphone strings also contain Googlebot/2.1 and a Chrome version that changes. Match the token, not one full string.
- IP list: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
- Source: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers (retrieved 2026-10-17)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/googlebot`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows Googlebot on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Googlebot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["googlebot"]}}}'
```

That call returns fetched_target false. This row is disallow via group Googlebot, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.