Where does a robots.txt file need to be hosted?

Definition: A robots.txt file contains rules about crawler access and must be available through the lowercase path /robots.txt at a website's root. Its scope is the website's origin: the combination of protocol, hostname and port.

Key facts

• RFC 9309, section 2.3, requires the top-level path /robots.txt and a text file encoded in UTF-8, a character encoding, served with the media type text/plain.
• As of 17 October 2026, Google's robots.txt specification documents rules that apply only within the file's protocol, host and port, including all subdirectories there.
• Google's creation guide recommends opening the file in a private browser window to check that it is publicly accessible before testing its rules.

What does the root location mean?

The root is the top-level URL path immediately after the hostname and any port. Google's creation guide gives this location for a site hosted at https://www.example.com/:

https://www.example.com/robots.txt

The same guide excludes placing the file in a subdirectory, such as https://example.com/pages/robots.txt. A website installed under a folder still needs its crawler rules at the host's root. This requirement concerns the publicly accessible URL; the guide directs site owners who need help accessing the root to their hosting provider.

Which website does that file control?

Google's scope examples show that https://example.com/robots.txt covers both https://example.com/ and https://example.com/folder/file. It does not cover https://other.example.com/, http://example.com/ or https://example.com:8181/.

The same documentation treats an explicit default port as equivalent to omitting that port: https://example.com:443/ has the same scope as https://example.com/. A nonstandard port, such as 8181, has its own scope. The practical check starts with the exact origin serving the pages whose crawling is being configured.

Can the root URL redirect elsewhere?

RFC 9309, section 2.3.1.2, recommends that crawlers follow at least five consecutive redirects, including redirects across hosts. When the file is reached within that chain, the standard requires its rules to be followed in the context of the initial authority, meaning the originally requested host.

A redirect can therefore lead to a file stored elsewhere while preserving the initial /robots.txt request location. The rules retain the scope of the initial request.

How can the location be checked?

Google's testing instructions start with opening the exact robots.txt URL in a private browser window and checking that its contents appear. This checks public access without relying on an existing signed-in browser session.

The same guide identifies the robots.txt report in Search Console for checking files already accessible on the site. Checking the URL and checking the rules are separate steps: a file can be publicly available while its rules still need correction.

Related terms

• robots.txt
• Do subdomains need separate robots.txt files?
• Do robots.txt rules apply across protocols and ports?
• HTTP redirects

FAQ

Can you put a separate robots.txt file in each folder?

No. Google's specification says crawlers do not check subdirectories for robots.txt files. Folder restrictions belong in the root file's rules.

Do you need to submit the file to Google?

Google's creation guide says its crawlers automatically find and use the file after it is uploaded and tested. No separate submission is required.

Sources

1. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
2. Google, "How Google interprets the robots.txt specification", updated 31 August 2026
3. Google, "How to write and submit a robots.txt file", updated 21 November 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer