`Cloudflare-AI-Search`

Cloudflare says AI Search indexes a website you own, in the same Cloudflare account, that you select as a data source. The crawler uses the user agent Cloudflare-AI-Search. The page says you can allow or block it with WAF rules using its bot detection ID, and does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare or Cloudflare-AI does not match, and it still reports the RFC 9309 decision.

robots.txt token: Cloudflare-AI-Search
Operator: Cloudflare
Kind: owner_requested
robots.txt: not_stated. The cited page does not say whether robots.txt controls this token.
HTTP user-agent: Cloudflare-AI-Search The page prints Cloudflare-AI-Search as the user agent. It does not print a separate robots.txt User-agent line. A group named Cloudflare does not match this token. A group named Cloudflare-AI does not match this token.
IP list: None on the cited page. The page does not publish an IP list. Botcard does not copy addresses.
Source: Cloudflare crawlers. Retrieved 2026-10-18, page says updated 2026-08-06.

JSON

curl -s https://whosthatbot.com/v1/crawlers/cloudflare-ai-search

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Cloudflare-AI-Search on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Cloudflare-AI-Search\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["cloudflare-ai-search"]}}}'

That call returns fetched_target false. This row is disallow via group Cloudflare-AI-Search, rule Disallow: /.