# Bytespider

Toutiao Search's webmaster page says Bytespider crawls pages, builds an index, and answers search queries. The page does not mention generative-model training or robots.txt. A separate Robots matching page (https://zhanzhang.toutiao.com/docs/intro/520) describes * and $ wildcards, path rules it says match Google's, and unordered query-parameter matching. That page does not name Bytespider and does not say the crawler honors robots.txt. The About page says a reverse-DNS hostname ending in .bytedance.com identifies the crawler, and that a crawl-frequency limit set in the webmaster tool takes effect within one day. Questions go to zhanzhang@bytedance.com.

- Operator: ByteDance
- Kind: search
- robots.txt: not_stated. The cited page does not say whether robots.txt controls this token.
- HTTP user-agent: Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36 Toutiao Search says the crawler user agent is Bytespider, capital B, and calls this the PC example. The same page also publishes Android and iOS examples that contain Bytespider and https://zhanzhang.toutiao.com/.
- IP list: none on the cited page
- Source: https://zhanzhang.toutiao.com/docs/intro/26899 (retrieved 2026-10-18)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/bytespider`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows Bytespider on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Bytespider\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["bytespider"]}}}'
```

That call returns fetched_target false. This row is disallow via group Bytespider, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.