Can I block Meta's AI crawler while allowing link previews?

Definition: Yes. As of 17 October 2026, Meta documents separate crawler tokens: meta-externalagent for uses including AI model training, and facebookexternalhit for shared-link previews, allowing separate robots.txt rules for each.

Key facts

• Meta's web crawler documentation identifies Meta-ExternalAgent as a crawler for training foundation AI models or improving products through direct content indexing. FacebookExternalHit gathers preview information for shared links.
• The IETF's Robots Exclusion Protocol specifies a website's root /robots.txt file and crawler-specific groups, which associate a user-agent token with access rules.
• Meta's sharing guide says Sharing Debugger shows the metadata scraped by the preview crawler, including errors and warnings, and triggers a scrape of the page.

Which published tokens serve which purposes?

A user-agent string identifies the crawler making a request. Meta's crawler documentation publishes distinct strings for these purposes:

• meta-externalagent collects content for uses including foundation model training and product improvement through indexing.
• facebookexternalhit gathers, caches and displays a shared page's title, description and thumbnail image on Meta apps such as Facebook, Instagram and Messenger.
• meta-webindexer crawls to improve Meta AI search result quality.
• meta-externalfetcher fetches individual links at a user's request and supports functions including evaluating and improving agentic AI capabilities.
• meta-externalads crawls for advertising and other business-related products and services.

The same documentation says Meta-ExternalFetcher may bypass robots.txt because its fetches are user-requested. FacebookExternalHit may also bypass the file for security or integrity checks. A rule targeting Meta-ExternalAgent therefore expresses a preference for that crawler's collection activity; search indexing and user-requested retrieval have separate published identities.

What robots.txt configuration separates training collection from previews?

Meta's robots.txt guidance instructs site owners to add a disallow rule for the relevant crawler. The following configuration applies that instruction to Meta-ExternalAgent and explicitly allows the preview crawler:

User-agent: meta-externalagent
Disallow: /

User-agent: facebookexternalhit
Allow: /

These groups belong in /robots.txt. Under RFC 9309, token matching is case-insensitive. A crawler uses the wildcard User-agent: * group when no matching group exists. An explicit preview group is therefore useful when the file already contains a wildcard restriction.

The same standard requires rules from multiple groups matching the same token to be combined. Existing groups for either token need review before merging the example into a site's file. More specific path rules can change the result for individual URLs. The example permits the preview crawler across the site, so any intended path restrictions need corresponding rules in its group.

Robots.txt communicates crawl instructions. RFC 9309 treats access security separately and names HTTP authentication as an access-control measure.

How can the preview be checked after changing the rules?

Meta's crawler documentation allows up to 24 hours for robots.txt changes to take effect because crawlers may cache the file. Its published log examples identify preview requests with facebookexternalhit/1.1 and training-related requests with meta-externalagent/1.1.

The sharing guide directs site owners to enter a page URL into Sharing Debugger to inspect scraped metadata and errors. It also describes testing the problem URL with the Facebook user-agent string to identify user-agent handling problems. This checks whether the preview crawler can retrieve the page after access rules change.

Related terms

• Meta-ExternalAgent
• facebookexternalhit
• Meta-ExternalFetcher
• Meta-WebIndexer
• robots.txt group precedence

FAQ

Which token should I disallow for training-related crawling?

Use meta-externalagent. Meta's crawler documentation names training foundation AI models among its purposes and instructs site owners to disallow the relevant crawler.

Do I need to allow Meta-WebIndexer to keep link previews?

Meta assigns shared-link previews to facebookexternalhit and Meta AI search indexing to meta-webindexer. Keep your rules for those published tokens separate according to the access you intend to allow.

How do I check whether Facebook can read my preview metadata?

Enter your page URL into Sharing Debugger. Meta's sharing guide says it displays scraped meta tags, errors and warnings, and triggers a new scrape.

Sources

1. Meta, "Meta Web Crawlers", accessed 17 October 2026
2. IETF, "Robots Exclusion Protocol", September 2022
3. Meta, "A Guide to Sharing for Webmasters", accessed 17 October 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer