`Googlebot-News`

Google says Googlebot News has no separate HTTP user-agent. Crawling uses various Googlebot strings. Preferences for Googlebot-News affect Google News, including news.google.com and the Google News app. The page also lists Googlebot as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named Googlebot does not match Googlebot-News, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.

robots.txt token: Googlebot-News
Operator: Google
Kind: search
robots.txt: obeys. The vendor says this crawler obeys robots.txt on automatic crawls.
Also matches: Googlebot. Google says one matching token is enough.
HTTP user-agent: The page says Googlebot-news does not have a separate HTTP request user agent string. Crawling is done with various Googlebot user agent strings. The robots.txt token spelling is Googlebot-News. A group named Googlebot does not match this token.
IP list: https://developers.google.com/static/crawling/ipranges/common-crawlers.json The common crawlers page says these crawlers generally crawl from this list, and that a user-agent string can be spoofed. Verify with reverse DNS before trusting an IP. The page's hostname mask is crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. URL returned HTTP 200 on 2026-10-18. Botcard does not copy the list.
Source: List of Google's common crawlers. Retrieved 2026-10-18, page says updated 2026-10-15. CC BY 4.0 (Google Developers page, except as that page notes).

JSON

curl -s https://whosthatbot.com/v1/crawlers/googlebot-news

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Googlebot-News on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Googlebot-News\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["googlebot-news"]}}}'

That call returns fetched_target false. This row is disallow via group Googlebot-News, rule Disallow: /. also_matches_tokens lists Googlebot and is not merged into this decision.