# Cloudflare-AI-Search

Cloudflare says AI Search indexes a website you own, in the same Cloudflare account, that you select as a data source. The crawler uses the user agent Cloudflare-AI-Search. The page says you can allow or block it with WAF rules using its bot detection ID, and does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare or Cloudflare-AI does not match, and it still reports the RFC 9309 decision.

- Operator: Cloudflare
- Kind: owner_requested
- robots.txt: not_stated. The cited page does not say whether robots.txt controls this token.
- HTTP user-agent: Cloudflare-AI-Search The page prints Cloudflare-AI-Search as the user agent. It does not print a separate robots.txt User-agent line. A group named Cloudflare does not match this token. A group named Cloudflare-AI does not match this token.
- IP list: none on the cited page
- Source: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/ (retrieved 2026-10-18)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/cloudflare-ai-search`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows Cloudflare-AI-Search on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Cloudflare-AI-Search\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["cloudflare-ai-search"]}}}'
```

That call returns fetched_target false. This row is disallow via group Cloudflare-AI-Search, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.