In short. Meta-WebIndexer crawls content for Meta AI search, and website owners can express access preferences through crawler-specific robots.txt rules.

Definition: Meta-WebIndexer is Meta’s web crawler for improving Meta AI search results. As of 17 October 2026, Meta documents robots.txt as the control for expressing whether this crawler may access a website’s content.

Key facts

Meta’s web crawler documentation connects Meta-WebIndexer with search result quality and links to website content in Meta AI responses.
Meta’s April 2024 announcement describes Meta AI search as a way to access real-time information from across the web inside its apps.
The IETF’s Robots Exclusion Protocol, RFC 9309, specifies crawler-specific rules in a public file at /robots.txt.

How does Meta-WebIndexer identify itself?

A user-agent string is the identifier a crawler sends in an HTTP request. Meta’s crawler documentation lists this short form for Meta-WebIndexer:

meta-webindexer/1.1

The same documentation says that the strings appearing in log files resemble its examples. Reviewing requests containing meta-webindexer is therefore a way to locate this declared crawler identity in server logs.

Meta describes its purpose as improving the relevance and accuracy of Meta AI through analysis of online content. It also connects allowing the crawler with using links to website content in responses. The documentation gives no citation probability for an accessible page.

Which robots.txt rules control it?

A robots.txt file publishes crawling preferences for a website. Meta’s robots guidance instructs site owners to add a disallow rule for the relevant crawler to block it. Applied to Meta-WebIndexer, that produces:

User-agent: meta-webindexer
Disallow: /

This example requests that the crawler avoid all paths. An alternative permits crawling except for a directory, adapting the directory example in the same documentation:

User-agent: meta-webindexer
Allow: /
Disallow: /private/

The RFC 9309 matching rules use a crawler’s product token, here meta-webindexer, to select its group. Matching the token is case-insensitive. The most specific matching path rule applies, so /private/ takes precedence over / in the second example.

The standard uses the wildcard group User-agent: * when there is no matching crawler-specific group. Checking both groups matters when reviewing an existing file. Meta asks for up to 24 hours for robots.txt changes to take effect because its crawlers may cache the file.

How does it relate to other Meta crawlers?

Meta’s crawler reference assigns model training and direct content indexing for product improvement to Meta-ExternalAgent. It assigns fetching individual links at a user’s request to Meta-ExternalFetcher. FacebookExternalHit gathers information such as titles and thumbnail images for shared links.

These names have separate purposes and user-agent identifiers. Under the standard’s group-matching rules, a group naming only meta-webindexer applies to that token. A website’s policy for Meta-ExternalAgent therefore requires its own applicable rules.

FAQ

How long should I wait after editing robots.txt?

Allow up to 24 hours, according to Meta’s crawler documentation. It attributes this delay to caching of robots.txt.

Does Disallow protect private content?

RFC 9309 identifies HTTP authentication as a measure for controlling access to protected paths. Use authentication for confidential content; robots.txt publishes crawling preferences and exposes the paths it lists.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer