In short. GPTBot collects content that may be used to train OpenAI models, with a robots.txt control separate from OpenAI's search crawler.

Definition: GPTBot is OpenAI's web crawler for collecting content that may be used to train its AI models. As of 17 October 2026, OpenAI documents a GPTBot-specific robots.txt control for indicating that site content should be excluded from training.

Key facts

OpenAI's crawler documentation identifies GPTBot as the crawler name used for training-related collection.
OpenAI documents independent settings for GPTBot and OAI-SearchBot, allowing a website to exclude training crawling while allowing search crawling.
OpenAI's GPTBot entry supplies an example user-agent string and a link to published IP address ranges.
The Robots Exclusion Protocol places crawler instructions in a file named /robots.txt at the top level of the service.

What changes for ChatGPT search?

OAI-SearchBot is OpenAI's crawler for surfacing websites in ChatGPT search results. OpenAI's crawler overview explicitly describes allowing OAI-SearchBot while disallowing GPTBot. A website therefore has separate settings for search crawling and training-related crawling.

The same documentation says that, when both crawlers are allowed, OpenAI may reuse the results of a single crawl for both purposes to avoid duplicate crawling. Server logs therefore need to be interpreted alongside the site's crawler settings.

How is GPTBot blocked in robots.txt?

A robots.txt file contains groups of rules addressed to crawler names. The Robots Exclusion Protocol defines a group as user-agent lines followed by rules, with the next user-agent line or the end of the file ending that group.

The following configuration expresses the separate choices described in OpenAI's documentation:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot selects GPTBot's group. Disallow: / excludes paths beginning at the root. The OAI-SearchBot group allows search crawling across those paths. The protocol's access rules require the file to be available at /robots.txt, encoded in UTF-8 and served as text/plain.

Existing rules matter when editing this file. Under the protocol's matching rules, groups matching the same crawler are combined. A matching crawler-specific group takes precedence over the fallback group named *.

How can GPTBot requests be checked?

A user-agent string identifies the client making a request. OpenAI provides this GPTBot example, whose version number may change:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

A practical log review checks requests containing GPTBot and compares their source IP addresses with the published ranges linked from OpenAI's crawler documentation. OpenAI also documents an optional robots.txt marker in the user-agent string when fetching that file, which helps distinguish rule fetches from other resource requests.

FAQ

Can I allow ChatGPT search crawling while blocking GPTBot?

Yes. OpenAI documents independent settings for GPTBot and OAI-SearchBot. You can use separate robots.txt groups to express those choices.

Does a GPTBot rule also control user-requested visits?

OpenAI identifies ChatGPT-User as the agent for certain user-initiated visits. Because users initiate those actions, OpenAI says robots.txt rules may not apply.

Does robots.txt protect confidential pages?

The Robots Exclusion Protocol states that its rules are not access authorization. You need access controls for confidential content.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer