In short. facebookexternalhit fetches shared content for Meta link previews, while Meta documents separate crawler identities for AI collection and search.

Definition: As of 17 October 2026, facebookexternalhit identifies Meta's FacebookExternalHit crawler, which fetches shared links and collects information for previews in apps such as Facebook, Instagram and Messenger. Meta documents separate crawlers for AI model training and AI search.

Key facts

Meta's Meta Web Crawlers documentation says FacebookExternalHit gathers, caches and displays information such as a shared page's title and thumbnail image.
Meta's sharing guide for webmasters describes Open Graph tags that specify how shared content appears on Facebook.
The IETF's Robots Exclusion Protocol defines crawler-specific groups in robots.txt, the file that expresses a site's crawl rules.

How does link preview fetching work?

Meta's crawler documentation describes a link shared by copying and pasting or through the Facebook social plugin. FacebookExternalHit fetches information about the linked app or website and caches it for display. A preview can include the page description alongside its title and thumbnail.

A user-agent string is the identifier a client sends with a request. Meta publishes these examples for FacebookExternalHit requests in server logs:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1

Open Graph properties are HTML metadata fields describing shared content. Meta's webmaster guide assigns the article title to og:title, its description to og:description, and the preview image URL to og:image. Without these tags, the crawler uses internal heuristics to select preview information.

How is this separate from AI collection?

Meta's crawler directory assigns different jobs to distinct crawler names. Meta-ExternalAgent crawls for uses including training foundation AI models and improving products through direct content indexing. Meta-WebIndexer crawls to improve Meta AI search results. Meta-ExternalFetcher retrieves individual links at a user's request and supports functions involving AI agents.

FacebookExternalHit's documented primary purpose is fetching shared content for previews. These distinct identifiers allow a website's crawl policy to address preview fetching separately from Meta-ExternalAgent collection.

What controls apply to previews?

The Meta documentation instructs site owners to add a disallow rule for the relevant crawler in robots.txt. The following configuration applies separate rules to preview fetching and Meta-ExternalAgent:

User-agent: facebookexternalhit
Allow: /

User-agent: meta-externalagent
Disallow: /

Under the IETF's group-selection rules, a crawler matches its product token to a user-agent group. Here, Allow: / permits preview crawling throughout the site, while the other group disallows Meta-ExternalAgent throughout the site.

Meta says FacebookExternalHit may bypass robots.txt for security or integrity checks, including checks for malware or malicious content. It also allows up to 24 hours for robots.txt changes to take effect because crawlers may cache the file for that period.

How can a website check a preview?

Meta's crawler requirements place Open Graph properties before the first 1 MB of content and require content to be crawlable within a few seconds. The server must support gzip and deflate encodings. For requests containing a Range header, it should return the required properties within the requested bytes or ignore that header altogether.

Meta's sharing guide recommends entering the page URL in Sharing Debugger. The tool shows scraped metadata and errors or warnings, and triggers another scrape. This provides a concrete check after correcting unavailable content or preview tags.

FAQ

Can I disallow Meta-ExternalAgent while allowing previews?

Yes. You can give facebookexternalhit and meta-externalagent separate robots.txt groups, following the distinct crawler identities and controls in Meta's documentation.

How can I inspect the information Facebook fetches?

You can submit your URL to Sharing Debugger. Meta's webmaster guide says it displays the meta tags scraped by the crawler and any errors or warnings.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer