In short. Meta-ExternalFetcher performs user-requested link fetches and may bypass robots.txt, so restricted resources need enforced access controls.

Definition: Meta-ExternalFetcher is Meta’s web crawler for fetching individual links at a user’s request. As of 17 October 2026, Meta documents that it may bypass robots.txt rules because these fetches are user requested.

Key facts

Meta’s web crawler documentation associates Meta-ExternalFetcher with evaluating and improving AI capabilities that help complete tasks on websites.
The IETF’s Robots Exclusion Protocol specifies robots.txt rules and recommends HTTP authentication as an access-security measure.
The IETF’s HTTP Semantics specification defines the 403 Forbidden status as a server’s refusal to fulfill a request it understands.

What triggers a fetch?

Meta’s crawler documentation describes the trigger as a user’s request for an individual link. It also describes support for evaluating and improving agentic AI capabilities, meaning capabilities that help AI complete tasks for users by navigating websites.

The same documentation assigns separate purposes to other named crawlers. Meta-ExternalAgent collects web content for uses such as training foundation AI models or improving products through direct content indexing. Meta-WebIndexer browses the web to improve Meta AI search results. These descriptions make the crawler name relevant when interpreting a request: the documented role of Meta-ExternalFetcher is user-directed fetching.

How does it identify itself?

A user-agent string is an identification string a client sends with a request. Meta’s published examples say that log entries for this fetcher resemble either of these forms:

meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-externalfetcher/1.1

The short form is itself a documented example. A review of requests can therefore look for the meta-externalfetcher name without requiring the parenthesized documentation path. These are identification examples from Meta’s documentation; the version shown belongs to those examples.

How should access be handled?

A robots.txt file expresses crawler instructions through rules such as Allow and Disallow. Meta’s robots.txt guidance states that Meta-ExternalFetcher may bypass those rules for user-requested fetches. A Disallow entry therefore provides an insufficient basis for treating a resource as protected from this fetcher.

For restricted content, the security considerations in RFC 9309 recommend an application-level security measure, with HTTP authentication as the example. They also warn that listing paths in robots.txt exposes those paths publicly.

Access handling can be checked against the server’s response. RFC 9110 defines 401 Unauthorized for a request lacking valid authentication credentials and requires a WWW-Authenticate challenge in that response. It defines 403 Forbidden for a request the server understands but refuses to fulfill. These are general HTTP response meanings that apply when reviewing access decisions for a URL.

FAQ

Can I block Meta-ExternalFetcher through robots.txt alone?

Meta’s documentation says it may bypass robots.txt because the user requests the fetch. For content requiring restricted access, use an enforced access control such as authentication.

Should I list confidential URLs in robots.txt?

The Robots Exclusion Protocol’s security section warns that listed paths become publicly visible. Protect confidential resources with application-level security controls.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer