OAI-SearchBotOpenAI says OAI-SearchBot is used to surface sites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says this setting is independent of GPTBot, and that a robots.txt change can take about 24 hours to affect search.
searchdisallow_is_opt_out. The vendor says disallowing this token opts that use out.Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot OpenAI calls this an example. The version number may change. A robots.txt fetch may add a robots.txt marker.curl -s https://whosthatbot.com/v1/crawlers/oai-searchbot
tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows OAI-SearchBot on /docs and allows *. The call does not fetch a host.
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: OAI-SearchBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["oai-searchbot"]}}}'
That call returns fetched_target false. This row is disallow via group OAI-SearchBot, rule Disallow: /.