# ClaudeBot

Anthropic says ClaudeBot collects public web content that may be used to train generative models. Restricting it signals that the site's future material should be left out of those training sets. The article says Anthropic's bots honor robots.txt, including a non-standard Crawl-delay.

- Operator: Anthropic
- Kind: training
- robots.txt: disallow_is_opt_out. The vendor says disallowing this token opts that use out.
- HTTP user-agent: The cited help article names the robots.txt token and does not publish a full HTTP user-agent string. Botcard leaves that field empty.
- IP list: https://claude.com/crawling/bots.json
- Source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (retrieved 2026-10-17)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/claudebot`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows ClaudeBot on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: ClaudeBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["claudebot"]}}}'
```

That call returns fetched_target false. This row is disallow via group ClaudeBot, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.