`Meta-ExternalFetcher`

Meta says Meta-ExternalFetcher fetches individual links at a person's request. It supports product functions such as evaluating and improving agentic AI, including helping AI navigate a site to complete tasks. Meta says this crawler may bypass robots.txt because a person requested the fetch. The checker still reports the RFC 9309 match and does not fetch a host. Meta-ExternalAgent and Meta-WebIndexer are different tokens.

robots.txt token: Meta-ExternalFetcher
Operator: Meta
Kind: user_fetch
robots.txt: may_not_apply. The vendor says robots.txt may not apply, because a person triggered the fetch.
HTTP user-agent: meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) Meta says the log string will be similar to this one or to meta-externalfetcher/1.1. On 2026-10-18 the page was text/markdown, and the parenthetical path has no host.
IP list: None on the cited page. This section does not publish an IP list.
Source: Meta Web Crawlers. Retrieved 2026-10-18.

JSON

curl -s https://whosthatbot.com/v1/crawlers/meta-externalfetcher

Check this token

tools/call with name check_robots is the same check at POST /mcp. The pasted file disallows Meta-ExternalFetcher on /docs and allows *. The call does not fetch a host.

curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: Meta-ExternalFetcher\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["meta-externalfetcher"]}}}'

That call returns fetched_target false. This row is disallow via group Meta-ExternalFetcher, rule Disallow: /.