# CCBot

Common Crawl's CCBot page says the crawler identifies itself as CCBot/2.0 with the FAQ URL inside the user-agent. The page description says it is a Nutch-based crawler that uses the crawl database to pick candidates. The body says the crawl supports an open repository of web crawl data, and that publishing User-agent: CCBot and Disallow: / prevents Common Crawl from crawling the site. The page says some crawlers falsely use the CCBot name, and that the real crawler uses dedicated IP ranges with reverse DNS, except IPv6 where reverse DNS is not yet supported. The CCBot page shows no article date. A separate post dated September 17, 2025 says the Opt-Out Registry is now the Opt-Out Ledger, because the list is immutable. That post says Common Crawl publishes every legal opt-out request it has received, encourages robots.txt to control crawling, and still processes legal requests. It points CCBot configuration back at the CCBot page and does not say the ledger replaces the Disallow block.

- Operator: Common Crawl
- Kind: open_crawl
- robots.txt: disallow_is_opt_out. The vendor says disallowing this token opts that use out.
- HTTP user-agent: CCBot/2.0 (https://commoncrawl.org/faq/) The cited page publishes this as the UserAgent string. The FAQ URL is inside the parentheses. The same page says some crawlers falsely identify themselves as CCBot and recommends verifying the string.
- Opt-out ledger: Opt-Out Ledger (was Opt-Out Registry). Post: https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry (header date 2025-09-17, retrieved 2026-10-18). Ledger: https://docs.google.com/spreadsheets/d/1uavIZ-Y2ew-Vj7d0_pY67SGD2jcUCsZZ5rPy1qEyPzg/edit?gid=0#gid=0. The post header shows September 17, 2025. The body says the Opt-Out Registry is now the Opt-Out Ledger, to reflect that the list is immutable. Common Crawl says it publishes the full list of legal opt-out requests it has received, encourages robots.txt to control crawling, and continues to process legal requests. Questions and requests go to info@commoncrawl.org. The post points CCBot configuration at the CCBot page. The checker reports this and does not apply it to the RFC decision.
- IP list: https://index.commoncrawl.org/ccbot.json
- Source: https://commoncrawl.org/ccbot (retrieved 2026-10-18)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/ccbot`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows CCBot on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: CCBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["ccbot"]}}}'
```

That call returns fetched_target false. This row is disallow via group CCBot, rule Disallow: /. opt_out_ledger is reported and is not the decision.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.