In short. Meta-ExternalAgent collects web content for uses including AI model training and product improvement, with crawler-specific robots.txt controls.

Definition: Meta-ExternalAgent is a web crawler, software that fetches web content, operated by Meta. As of 17 October 2026, Meta documents its uses as including AI model training and product improvement through direct content indexing.

Key facts

Meta's web crawler documentation lists model training and product improvement as uses of Meta-ExternalAgent.
The HTTP standard, RFC 9110, defines a User-Agent header that identifies the software making a request through product names and optional versions.
The Robots Exclusion Protocol, RFC 9309, defines Allow and Disallow rules in robots.txt, a file that publishes instructions for crawlers about permitted paths.

What does Meta collect content for?

A web crawler is software that fetches content from websites or web applications. Meta's crawler documentation describes Meta-ExternalAgent as crawling for uses such as training AI models and improving products through direct content indexing. Its stated scope therefore includes collection for more than model training alone.

The same documentation assigns Meta-WebIndexer the task of improving Meta AI search result quality. For a website owner, the distinction matters when choosing which crawler a robots.txt rule addresses: meta-externalagent targets the collection crawler described for training and product improvement, while meta-webindexer identifies the search indexing crawler.

How can a website control Meta-ExternalAgent?

A robots.txt file publishes instructions for crawlers. Meta's configuration guidance tells site managers to add a Disallow rule for the relevant crawler. Its example allows crawling generally while excluding a directory:

User-agent: meta-externalagent
Allow: /
Disallow: /private/

For a site-wide exclusion, the path in the disallow rule becomes /:

User-agent: meta-externalagent
Disallow: /

Under RFC 9309, crawlers match their product token to a user-agent group without regard to letter case. The most specific matching path rule applies. In the directory example, /private/ takes precedence over the broader / allowance.

Meta states that robots.txt changes can take up to 24 hours to take effect because crawlers may cache the file for that long. It recommends robots.txt for expressing preferences and describes NoAI tags as a non-standard format.

How does the crawler identify itself?

A user-agent string identifies the software making an HTTP request. Meta's published log examples include this short form:

meta-externalagent/1.1

The robots.txt group uses meta-externalagent without the version suffix. This connects the identifier seen in logs with the rule used to express crawling preferences.

The HTTP standard, RFC 9110, also discusses software presenting itself as a different user agent. A matching header is therefore a declared identity; assessing whether requests actually originate from Meta requires additional evidence.

FAQ

Does a Meta-ExternalAgent rule cover only training?

Meta's documentation describes both training and product improvement for this crawler. A rule addressed to its token applies to the crawler with that documented scope.

How soon should a robots.txt change take effect?

Meta specifies up to 24 hours because its crawlers may cache robots.txt. You can compare subsequent requests with the paths covered by the revised rule after that interval.

Can robots.txt protect confidential pages?

RFC 9309 states that robots.txt rules are not access authorization. You need access controls for confidential content, even when its paths appear in a Disallow rule.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer