`Diffbot`

Diffbot says its web crawls follow a site's robots.txt by default, including Disallow and crawl-delay, and that a partnership or agreement with the site can override that. This check does not apply that override and does not apply crawl-delay. Diffbot says this user agent is general proactive crawling so sites can be discovered and cited from the Diffbot Knowledge Graph and web search, and that it is not used for AI training. Diffbot-User is a different token. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.

robots.txt token: Diffbot
Operator: Diffbot
Kind: search
robots.txt: obeys_unless_agreed. The vendor says automatic crawls follow robots.txt, including Disallow and crawl-delay, unless a partnership or agreement overrides that. This check does not apply that override.
HTTP user-agent: The page lists Diffbot in a User-Agent column and does not publish a full HTTP user-agent string. The Similar To column says Googlebot, Bing.
IP list: None on the cited page. This page does not publish an IP list.
Source: Does Diffbot respect robots.txt?. Retrieved 2026-10-18, page says updated 2026-07-27.

JSON

curl -s https://whosthatbot.com/v1/crawlers/diffbot

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Diffbot on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Diffbot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["diffbot"]}}}'

That call returns fetched_target false. This row is disallow via group Diffbot, rule Disallow: /.