Storebot-GoogleGoogle says preferences for Storebot-Google affect all surfaces of Google Shopping, including the Shopping tab in Google Search and Google Shopping. The page lists only the token Storebot-Google. A group named Storebot does not match. This check matches the whole token and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
searchobeys. The vendor says this crawler obeys robots.txt on automatic crawls.Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 The page prints this desktop string and a mobile string: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36. Chrome/W.X.Y.Z is the page's placeholder for the Chrome version. The robots.txt token is Storebot-Google. The page does not list Googlebot on this row. A group named Storebot does not match.curl -s https://whosthatbot.com/v1/crawlers/storebot-google
tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Storebot-Google on /docs and allows *. The call does not fetch a host.
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Storebot-Google\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["storebot-google"]}}}'
That call returns fetched_target false. This row is disallow via group Storebot-Google, rule Disallow: /.