How do I block Mistral training while allowing search?

Definition: As of 17 October 2026, Mistral documents separate user-agent tokens, names used to match crawler rules, for training collection, search indexing and user-requested access. A robots.txt file, which specifies crawler access rules, can disallow MistralAI-Training while allowing MistralAI-Index and MistralAI-User independently.

Key facts

• Mistral's crawler documentation assigns training collection to MistralAI-Training, search indexing to MistralAI-Index, and user-requested page visits in Vibe to MistralAI-User.
• The IETF's Robots Exclusion Protocol specifies matching rules by user-agent group; the wildcard group applies when no named group matches.
• Google's robots.txt creation guide explains that the file applies within the protocol, host and port where it is posted.

Which rules express this policy?

The following configuration combines Mistral's published tokens with the group syntax in RFC 9309:

User-agent: MistralAI-Training
Disallow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-User
Allow: /

Each User-agent line starts a separate group. Disallow: / blocks crawling across the site's paths; Allow: / permits it. This example allows every path for both the indexer and the user-requested fetcher.

Mistral states that webmasters can disallow its training crawler through this file. The same documentation says the training crawler serves neither search indexing nor live queries, and that content collected by the indexer is excluded from model training.

How do existing rules affect the result?

Under RFC 9309's group matching rules, a named matching group supplies the applicable rules. A User-agent: * group supplies the fallback when no named group matches. Consequently, adding explicit groups for the search tokens permits their access even when the wildcard group disallows crawling.

Existing path restrictions need review: restrictions present only in the wildcard group are absent from these named groups. Any paths intended to remain excluded for search belong in the applicable named groups too.

Multiple groups matching the same token merge. Within the resulting rules, the most specific matching path applies; an equivalent allow and disallow should resolve to allow. An existing, more specific Allow path in a matching training group can therefore leave that path accessible despite Disallow: /. Reviewing the complete file matters when adding the training block.

How can the published file be checked?

Google's creation guide places robots.txt at the host root and requires a UTF-8 text file. Its accessibility check consists of opening the file's address in a private browser window and confirming that the contents appear.

The guide uses this address pattern:

https://example.com/robots.txt

For a site with several hosts or protocols, the same guide limits each file's scope to its own protocol, host and port. A copy on one subdomain therefore leaves another subdomain's rules separate. Comparing the served file with the intended groups checks what crawlers can read after deployment.

Related terms

• MistralAI-Training
• MistralAI-Index
• MistralAI-User
• robots.txt
• robots.txt group precedence

FAQ

Do I need to block MistralAI-User to refuse training collection?

Mistral's documentation identifies MistralAI-Training as the training collection token. You can target that token while leaving the user-requested access group allowed.

Can I keep some paths excluded while allowing search?

Yes. Under RFC 9309, you can put path-specific Disallow rules in the matching search groups. More specific path rules take precedence over Allow: /.

Sources

1. Mistral AI, "Mistral crawlers", undated
2. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
3. Google, "How to write and submit a robots.txt file", undated

---

Check how a page reads to answer engines. Open the GEO score analyzer