`Bytespider`

Toutiao Search's webmaster page says Bytespider crawls pages, builds an index, and answers search queries. The page does not mention generative-model training or robots.txt. A separate Robots matching page (https://zhanzhang.toutiao.com/docs/intro/520) describes * and $ wildcards, path rules it says match Google's, and unordered query-parameter matching. That page does not name Bytespider and does not say the crawler honors robots.txt. The About page says a reverse-DNS hostname ending in .bytedance.com identifies the crawler, and that a crawl-frequency limit set in the webmaster tool takes effect within one day. Questions go to zhanzhang@bytedance.com.

robots.txt token: Bytespider
Operator: ByteDance
Kind: search
robots.txt: not_stated. The cited page does not say whether robots.txt controls this token.
HTTP user-agent: Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36 Toutiao Search says the crawler user agent is Bytespider, capital B, and calls this the PC example. The same page also publishes Android and iOS examples that contain Bytespider and https://zhanzhang.toutiao.com/.
IP list: None on the cited page. The About page lists ten IPv4 prefixes and a reverse-DNS check whose hostname ends in .bytedance.com. Botcard does not copy the prefixes.
Source: 关于Bytespider. Retrieved 2026-10-18.

JSON

curl -s https://whosthatbot.com/v1/crawlers/bytespider

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Bytespider on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Bytespider\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["bytespider"]}}}'

That call returns fetched_target false. This row is disallow via group Bytespider, rule Disallow: /.