`Claude-SearchBot`

Anthropic says Claude-SearchBot crawls to improve search-result quality. Disabling it stops indexing for that search use and may reduce how visible or accurate the site is in search answers. It is a different token from ClaudeBot (training) and Claude-User (a person asked).

robots.txt token: Claude-SearchBot
Operator: Anthropic
Kind: search
robots.txt: disallow_is_opt_out. The vendor says disallowing this token opts that use out.
HTTP user-agent: The cited help article names the robots.txt token and does not publish a full HTTP user-agent string.
IP list: https://claude.com/crawling/bots.json Shared Anthropic crawler IP list linked from the article. URL returned HTTP 200 on 2026-10-17. Botcard does not copy the address list.
Source: Does Anthropic crawl data from the web, and how can site owners block the crawler?. Retrieved 2026-10-17.

JSON

curl -s https://whosthatbot.com/v1/crawlers/claude-searchbot

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Claude-SearchBot on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["claude-searchbot"]}}}'

That call returns fetched_target false. This row is disallow via group Claude-SearchBot, rule Disallow: /.