In short. Use a GPTBot group with Disallow: / and a separate OAI-SearchBot group with Allow: /, then check existing rules and server access.
Definition: Blocking GPTBot while allowing OAI-SearchBot uses separate groups in robots.txt, a file that declares crawler access rules. As of 17 October 2026, OpenAI documents these training and search settings as independent.
Key facts
- OpenAI's crawler documentation assigns GPTBot to crawling content potentially used for model training and OAI-SearchBot to ChatGPT search.
- The IETF's Robots Exclusion Protocol requires crawlers to combine rules from groups matching the same user-agent name.
- OpenAI publishes search crawler network ranges as
ipv4Prefixentries in its searchbot.json file.
Which robots.txt rules separate the crawlers?
OpenAI's crawler documentation explicitly supports allowing OAI-SearchBot while disallowing GPTBot. The following configuration implements that choice using the group and path syntax in the IETF's Robots Exclusion Protocol:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent identifies the crawler to which the following rules apply. Disallow: / blocks access to paths beginning with /, while Allow: / permits them. Each crawler has its own group because the next User-agent line after a rule starts another group.
The same protocol requires the file at the service's top-level /robots.txt path, encoded as UTF-8 and served as text/plain. The example is an all-path policy for each named crawler. Existing path restrictions need review before adding an all-path allowance.
How do existing groups affect this configuration?
Under the IETF's group-matching rules, the wildcard group, User-agent: *, applies when a crawler has no matching named group. An explicit OAI-SearchBot group therefore supplies its own rules. Restrictions in a wildcard group need to be repeated in that named group if they are intended to apply to OAI-SearchBot too.
Multiple groups naming OAI-SearchBot are combined. Adding a new group to the bottom of the file does not erase an earlier matching group. Within the combined rules, the longest matching path takes precedence; for equivalent allow and disallow rules, the protocol says the allow rule should prevail. This makes reviewing every matching group necessary when preserving restrictions on particular paths.
Putting both crawler names above a shared Disallow: / places them in the same group and applies the block to both. Separate rules beneath each name express the independent policies.
What should be checked after editing the file?
OpenAI's crawler guidance recommends permitting both OAI-SearchBot in robots.txt and requests from its published IP ranges. Server or firewall access rules therefore need review alongside the file. OpenAI says search systems can take approximately 24 hours to adjust after a robots.txt update.
The IETF's access requirements provide a concrete deployment check: retrieve the public /robots.txt and verify that it serves the intended plain-text rules. The protocol describes robots.txt as a crawler instruction mechanism; it recommends application-layer security measures for protecting restricted content.
Related terms
FAQ
Can I keep search crawling while declining GPTBot training crawling?
Yes. OpenAI's crawler documentation explicitly describes that combination. Separate GPTBot and OAI-SearchBot groups express your choices for each purpose.
Do I need to include a browser string or crawler version?
Use the crawler's name in the User-agent line. The IETF's user-agent specification distinguishes this product token from the longer identification string sent in an HTTP request.
Does this also control ChatGPT-User requests?
OpenAI's same crawler documentation describes ChatGPT-User as handling user-initiated actions and says robots.txt rules may not apply to those requests. Use OAI-SearchBot to manage automatic search crawling.
Sources
- OpenAI, "Overview of OpenAI Crawlers", undated, accessed 17 October 2026
- IETF, "Robots Exclusion Protocol", RFC 9309, September 2022
- OpenAI, OAI-SearchBot published IP addresses, accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer