# Diffbot-User

Diffbot says Diffbot-User is for requests on behalf of a person browsing a URL in Diffbot software, in response to that person's input. The page does not say whether this token obeys robots.txt. The default adherence sentence on the same page is about Diffbot web crawls, which are the Diffbot token, not this one. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.

- Operator: Diffbot
- Kind: user_fetch
- robots.txt: not_stated. The cited page does not say whether robots.txt controls this token.
- HTTP user-agent: The page lists Diffbot-User in a User-Agent column and does not publish a full HTTP user-agent string. The Similar To column says Mozilla, AppleWebKit.
- IP list: none on the cited page
- Source: https://www.diffbot.com/docs/crawl/faq/robots-txt (retrieved 2026-10-18)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/diffbot-user`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows Diffbot-User on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Diffbot-User\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["diffbot-user"]}}}'
```

That call returns fetched_target false. This row is disallow via group Diffbot-User, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.