# MojeekBot

Mojeek says MojeekBot is the web crawler for the Mojeek search engine. It obeys the Robot Exclusion Standard by taking the first record whose User-Agent contains MojeekBot, otherwise the first User-Agent * record, and it does not support the crawl-delay directive. This check matches the whole token and merges every group that names it, which is RFC 9309, and it does not apply crawl-delay. The page also says a noindex meta tag is fetched but not added to the search database. This check does not read meta tags.

- Operator: Mojeek
- Kind: search
- robots.txt: respects_no_crawl_delay. The vendor says this crawler respects robots.txt on general search crawls aimed at this token, and does not follow crawl-delay.
- HTTP user-agent: The page names the robots.txt token MojeekBot and does not publish a full HTTP user-agent string.
- IP list: https://www.mojeek.com/mojeekbot.json
- Source: https://www.mojeek.com/bot.html (retrieved 2026-10-18)

## JSON

`curl -s https://whosthatbot.com/v1/crawlers/mojeekbot`

## Check this token

tools/call with name check_robots is the same check at https://whosthatbot.com/mcp. Send `Content-Type: application/json` and `Accept: application/json, text/event-stream`. The pasted file disallows MojeekBot on `/docs` and allows `*`. The call does not fetch a host.

```
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: MojeekBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["mojeekbot"]}}}'
```

That call returns fetched_target false. This row is disallow via group MojeekBot, rule Disallow: /.

Each card paraphrases the linked vendor page and was checked on source.retrieved. This is not legal advice. robots.txt is a request from the site owner. Botcard does not fetch third-party sites and does not suggest ways around a disallow rule.