In short. Robots.txt rules have origin-specific scope, with default ports treated as equivalent to omitted ports and redirects retaining the original request context.
Definition: A robots.txt file supplies crawl rules for one origin, the combination of protocol, hostname and port. HTTP and HTTPS have separate rules; an explicitly written default port is equivalent to omitting that port.
Key facts
- As of October 2026, Google's robots.txt specification limits a file's scope to its host, protocol and port number.
- Google's file creation guide places robots.txt at the root of the host to which it applies.
- RFC 9309, section 2.3.1.2 permits redirects across hosts while retaining the context of the initial authority, meaning the server originally addressed.
Which protocols and ports share rules?
Google's scope examples separate these robots.txt locations:
http://example.com/robots.txt
https://example.com/robots.txt
https://example.com:8181/robots.txt
The HTTPS file without a port applies to https://example.com/folder/file. The HTTP version and content on port 8181 have their own robots.txt locations.
The same documentation treats the default ports, 80 for HTTP and 443 for HTTPS, as equivalent to their omitted forms. Thus https://example.com:443/robots.txt covers both https://example.com:443/ and https://example.com/. It does not cover https://example.com:444/.
Hostnames also matter: the file at https://www.example.com/robots.txt applies to that hostname, with separate scope from https://example.com/.
Can a redirect supply the same file to another origin?
RFC 9309, section 2.3 defines the fetch location as scheme:[//authority]/robots.txt. Its redirect rules say crawlers should follow at least five consecutive redirects, including across authorities.
If a robots.txt file is reached within five redirects, the standard requires its rules to be followed in the context of the initial authority. A redirected fetch can therefore supply rules for the originally requested origin. The destination file's hosting location does not replace that original context.
This distinction matters when HTTP and HTTPS share a robots.txt file through a redirect. The relevant check starts at each origin's /robots.txt address and follows its response.
How can the deployed files be checked?
Google's testing instructions recommend opening the robots.txt address in a private browsing window to check public accessibility. Applying that check to each protocol and non-standard port follows the guide's stated scope rule.
The guide requires the filename robots.txt at the root. A copy inside /pages/ cannot serve as the host's robots.txt file. It also specifies a UTF-8 encoded text file.
After checking accessibility, the guide points to the robots.txt report in Search Console for markup issues. That report requires a file already accessible on the site.
Related terms
- robots.txt
- Where does a robots.txt file need to be hosted?
- Do subdomains need separate robots.txt files?
- HTTP redirects
FAQ
Does an HTTPS robots.txt file automatically control HTTP?
No. Google's scope rule separates protocols. Check your HTTP origin's /robots.txt response, including any redirect that supplies its rules.
Does adding :443 create separate HTTPS rules?
No. Google's examples treat HTTPS port 443 as equivalent to the default HTTPS address. A non-standard port such as 8181 has separate scope.
Must every origin store a separate physical copy?
RFC 9309 permits redirects across authorities. You can serve rules through a redirected robots.txt request, with the resulting rules applied in the context of the original authority.
Sources
- Google, "How Google interprets the robots.txt specification", updated 31 August 2026
- Google, "Create and submit a robots.txt file", updated 21 November 2025
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
Check how a page reads to answer engines. Open the GEO score analyzer