How do I publish robots.txt on Netlify?

Definition: Publishing a static robots.txt file on Netlify means including a plain text crawler rules file at the root of the publish directory, the folder Netlify deploys. The resulting public path is /robots.txt.

Key facts

• As of October 2026, Netlify's build configuration documentation states that only files inside the publish directory enter a site deploy.
• Netlify's custom headers documentation places the _headers configuration file in that same publish directory.
• Netlify's rewrite documentation explains that existing static files take precedence over ordinary rewrite rules.
• As of October 2026, Google's robots.txt specification guidance says its crawlers fetch the file using an HTTP GET request.

Where does the static file belong?

Netlify's build configuration overview defines the publish directory as the folder containing deploy-ready files. Its path is relative to the base directory. The publish property under [build] in netlify.toml sets this directory; configuration file settings override the corresponding interface settings.

The deployment requirement is that robots.txt ends up directly inside that directory after the build. A file kept elsewhere in the repository needs a build step or framework mechanism that copies it into the published output.

Google's file creation guide requires the name robots.txt and a UTF-8 plain text file at the host's root. This example uses its documented allow rule:

User-agent: *
Allow: /

This group allows crawling for the agents it matches. Existing restrictions need to be retained when that is the intended policy.

Can an application rewrite intercept robots.txt?

A rewrite serves another resource while keeping the requested URL. Netlify's rewrite documentation shows this fallback for a single-page application, an app that routes page navigation through its main HTML file:

/*  /index.html  200

The documentation places this rule last when other routing rules exist. An existing static /robots.txt survives this ordinary fallback. An exclamation mark, as in 200!, forces a matching rewrite even when a static file exists.

If /robots.txt displays the app, the routing checks cover a missing deployed file and forced rules matching that path. A successful response status alone cannot distinguish the intended rules file from rewritten HTML.

How are response headers configured?

An HTTP response header describes the returned content or its handling. Netlify's custom headers documentation defines a path followed by indented header lines in _headers. This path-specific example follows that syntax:

/robots.txt
  Content-Type: text/plain

The file belongs in the publish directory. Netlify also accepts header tables in netlify.toml.

These custom response headers apply to files served from Netlify's own backing store. For a route handled by a function, an edge function or a proxy, the function or upstream service must return the required headers.

RFC 9309, section 2.3, specifies UTF-8 encoding and the text/plain media type for robots.txt. The header configuration supplements a correctly deployed text file.

How is the deployed response checked?

Google's testing instructions recommend opening the public robots.txt URL in a private browser window. After deployment, the production host's /robots.txt response is compared with the deployed file.

The response check covers the status, any redirect destination and the Content-Type header alongside the body. Google's specification guidance says successful 2xx responses prompt parsing; if the body is HTML, Google tries to extract valid rules and ignores other content. This makes checking the actual response body essential when an application fallback is configured.

Related terms

• robots.txt
• HTTP status codes
• HTTP redirects
• X-Robots-Tag

FAQ

Can I leave robots.txt in the repository root?

Only if your build puts it into the publish directory, or that root is itself the publish directory. Netlify's build documentation excludes files outside the published output.

Do I need a function to serve the file?

You can deploy it as a static file. Netlify's headers documentation supports configuring response headers for files in its backing store.

How can I check whether Google parses my file?

After you make the file publicly accessible, you can use the robots.txt report in Search Console. Google's testing guide identifies this report as an option for checking markup issues.

Sources

1. Netlify, "Custom headers", 1 December 2025
2. Netlify, "Build configuration overview", 17 September 2026
3. Netlify, "Rewrites and proxies", 14 July 2025
4. Google, "How to write and submit a robots.txt file", 21 November 2025
5. Google, "How Google interprets the robots.txt specification", 31 August 2026
6. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer