In short. OpenAI says robots.txt rules may not apply to ChatGPT-User requests because a user initiates them.
Definition: ChatGPT-User identifies certain user-initiated web requests from ChatGPT. As of 17 October 2026, OpenAI's crawler documentation says robots.txt rules may not apply to these requests because a user initiates them.
Key facts
- OpenAI's crawler documentation distinguishes ChatGPT-User from automatic crawling and directs search opt-outs to OAI-SearchBot.
- The Robots Exclusion Protocol defines robots.txt rules and states that they are not access authorization.
- HTTP Semantics defines an authentication challenge that a server can send when a request lacks valid credentials.
What does OpenAI's qualification cover?
OpenAI's Overview of OpenAI Crawlers says a question from a user can lead ChatGPT to visit a web page using ChatGPT-User. The documented reason for the robots.txt qualification is that the user initiates the action. OpenAI uses the wording “robots.txt rules may not apply”.
For a website, the practical consequence is that a ChatGPT-User disallow rule cannot be relied on to prevent a user-directed visit. OpenAI identifies OAI-SearchBot as the control for automatic search crawling. It also says ChatGPT-User does not determine whether content may appear in Search.
How does robots.txt differ from access control?
A robots.txt file expresses rules for crawlers. RFC 9309, section 2.3, places this file at the service's top-level /robots.txt path and requires UTF-8 encoding and the text/plain media type. Section 2.2.1 tells crawlers to match their product token to a User-agent group and obey that group's rules.
The same specification's security section explains that listing paths in robots.txt exposes those paths publicly. It recommends a security measure appropriate to the application, giving HTTP authentication as an example for controlling access to protected content.
HTTP authentication lets a server challenge a request for credentials. RFC 9110, section 15.5.2, defines 401 Unauthorized as the response when a request lacks valid authentication credentials for the resource. That response must include a WWW-Authenticate header containing an applicable challenge. A site's private content therefore needs access controls enforced by its server, independently of its crawler preferences.
What identifies a ChatGPT-User request?
A user-agent string identifies the software making a request. OpenAI's crawler documentation publishes this full string for ChatGPT-User:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
The same documentation links to https://openai.com/chatgpt-user.json for its published IP addresses. These identifiers provide concrete references when reviewing requests associated with ChatGPT-User.
Related terms
FAQ
Can I use robots.txt to keep private pages private?
Private pages need server-enforced access controls. RFC 9309, section 3, recommends application-level security measures and specifically names HTTP authentication.
Which user agent controls automatic ChatGPT search crawling?
OpenAI's crawler documentation directs site owners to use OAI-SearchBot in robots.txt for search opt-outs and automatic crawling. Its separate ChatGPT-User qualification concerns user-initiated actions.
Sources
- OpenAI, "Overview of OpenAI Crawlers", accessed 17 October 2026
- IETF, "Robots Exclusion Protocol", September 2022
- IETF, "HTTP Semantics", June 2022
Check how a page reads to answer engines. Open the GEO score analyzer