# Diffbot

Diffbot says its web crawls follow a site's robots.txt by default, including Disallow and crawl-delay, and that a partnership or agreement with the site can override that. This check does not apply that override and does not apply crawl-delay. Diffbot says this user agent is general proactive crawling so sites can be discovered and cited from the Diffbot Knowledge Graph and web search, and that it is not used for AI training. Diffbot-User is a different token. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.

- Operator: Diffbot
- Kind: search
- robots.txt: obeys_unless_agreed. The vendor says automatic crawls follow robots.txt, including Disallow and crawl-delay, unless a partnership or agreement overrides that. This check does not apply that override.
- HTTP user-agent: The page lists Diffbot in a User-Agent column and does not publish a full HTTP user-agent string. The Similar To column says Googlebot, Bing.
- IP list: none on the cited page
- Source: https://www.diffbot.com/docs/crawl/faq/robots-txt (retrieved 2026-10-18)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/diffbot`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows Diffbot on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Diffbot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["diffbot"]}}}'
```

That call returns fetched_target false. This row is disallow via group Diffbot, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.