In short. Each subdomain needs its own robots.txt URL when crawl restrictions are required, with rules scoped to the host, protocol, and port.
Definition: As of 17 October 2026, Google documents robots.txt as a file of crawler access rules scoped to its host, protocol, and port. Each subdomain needs its own robots.txt URL when crawl restrictions are required.
Key facts
- Google's robots.txt specification limits a subdomain's file to that subdomain:
www.example.comhas a separate scope fromexample.com. - Google's file creation guide states that files are implicitly allowed for crawling unless rules specify otherwise.
- The IETF's Robots Exclusion Protocol permits redirects across hosts while retaining the scope of the initial host when the file is reached within five consecutive redirects.
Where does each robots.txt file belong?
A host is the hostname in a URL, such as www.example.com. Google's location and validity documentation places robots.txt in the host's top-level directory. For the following hosts, the file locations are:
https://example.com/robots.txt
https://www.example.com/robots.txt
The file at the first URL controls paths under https://example.com/. The file at the second controls paths under https://www.example.com/. The same documentation excludes nested subdomains such as shop.www.example.com from the scope of the file on www.example.com.
A subdirectory is a path below the host, such as /folder/. Its pages fall within the host's robots.txt scope. A file at https://example.com/folder/robots.txt is outside the location crawlers check.
Protocol and port also matter. HTTP and HTTPS have separate scopes. Google treats the explicit standard HTTPS port 443 as equivalent to the default HTTPS URL, while a nonstandard port such as 8181 has its own scope.
How are rules applied on a subdomain?
A User-agent line identifies the crawler targeted by a group of rules. A Disallow line specifies a path that crawler must avoid. Google's creation guide gives this group for blocking Googlebot:
User-agent: Googlebot
Disallow: /
When served at https://www.example.com/robots.txt, this group blocks Googlebot from crawling paths on that HTTPS host. Applying that restriction to https://example.com/ requires rules available through its robots.txt URL too. An instruction in the parent domain's file supplies no inherited policy for a subdomain.
The same guide requires a plain text file encoded in UTF-8. Hosting arrangements can affect how it is edited: a hosting provider may expose search settings instead of direct file access.
How can the files be checked?
Google's testing instructions recommend opening the robots.txt URL in a private browsing window to check public accessibility. Repeat that check for each host and protocol requiring rules, using the complete URL for each file.
The same instructions identify the robots.txt report in Search Console as a way to check markup issues for files already accessible on the site. A visible file establishes accessibility; its contents still need to express the intended crawler and path rules.
Related terms
FAQ
Does www count as a separate subdomain?
Yes. Google's validity examples explicitly separate https://www.example.com/ from https://example.com/.
Can I use a redirect to share a rules file?
The IETF's protocol specification allows redirects across hosts. When a file is reached within five consecutive redirects, its rules apply in the context of the original host whose robots.txt was requested.
Does blocking a subdomain keep its content private?
Use an access control such as HTTP authentication for private content. The same protocol's security considerations explain that robots.txt rules are publicly accessible and identify authentication as an appropriate security measure.
Sources
- Google Search Central, "How Google interprets the robots.txt specification", 31 August 2026
- Google Search Central, "How to write and submit a robots.txt file", 21 November 2025
- IETF, "Robots Exclusion Protocol", September 2022
Check how a page reads to answer engines. Open the GEO score analyzer