`Google-Extended`

Google says publishers use the Google-Extended robots.txt token to manage whether content Google has crawled may train future Gemini models (Gemini Apps and Vertex AI API for Gemini) and whether it may ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says this token does not change inclusion or ranking in Google Search.

robots.txt token: Google-Extended
Operator: Google
Kind: control_token
robots.txt: control_token. This is a robots.txt control. It is not a separate HTTP crawler.
HTTP user-agent: Google says Google-Extended has no separate HTTP user-agent. Crawling uses existing Google user-agent strings. The token is a robots.txt control.
IP list: None on the cited page. No separate crawler, so no separate IP list on the source page.
Source: List of Google's common crawlers. Retrieved 2026-10-17, page says updated 2026-10-15. CC BY 4.0 (Google Developers page, except as that page notes).

JSON

curl -s https://whosthatbot.com/v1/crawlers/google-extended

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Google-Extended on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Google-Extended\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["google-extended"]}}}'

That call returns fetched_target false. This row is disallow via group Google-Extended, rule Disallow: /.