In short. Edit custom robots.txt directives in Webflow under Site settings > SEO > Indexing, then save and publish the site.
Definition: As of 17 October 2026, Webflow's robots.txt editor is in Site settings > SEO > Indexing. A robots.txt file gives web crawlers instructions about which pages they may access; custom rules take effect after saving and publishing the site.
Key facts
- Webflow's robots.txt instructions place custom rules under Site settings > SEO > Indexing.
- Webflow's indexing documentation states that a page blocked through robots.txt remains in the automatically generated sitemap.
- Google's robots.txt introduction explains that a blocked URL can still appear in search results when other pages link to it.
How are custom rules saved?
Webflow's Set robots.txt rules guide gives the workflow: open Site settings > SEO > Indexing, add the desired rules, click Save, and publish the site. Saving is followed by publication before checking the live result.
The file lives at the domain's root, at a path such as yourdomain.com/robots.txt.
What does a rule look like?
Webflow's rule examples include this instruction for a folder:
User-agent: *
Disallow: /secret-folder/
The User-agent line identifies the crawler; * covers all bots. Disallow specifies a blocked path. Here, /secret-folder/ is Webflow's example folder. A Disallow: / line applies to the entire site, so the path determines the scope of the restriction.
How is the published file checked or reset?
After publication, opening the domain's /robots.txt address shows the file at the location specified in Webflow's documentation. Comparing that file with the saved directives checks what is publicly served.
Webflow's troubleshooting instructions state that an already created robots.txt file cannot be completely removed. To allow crawling again, the documentation gives these replacement rules:
User-agent: *
Disallow:
The empty Disallow value leaves no blocked path in this rule. The documented procedure is to save the replacement and republish. If the old rules persist on the published site, Webflow directs site owners to customer support.
Related terms
- robots.txt
- noindex
- How do I disable search indexing for a Webflow page?
- How do I keep a webflow.io staging site out of search?
FAQ
Does blocking a page remove it from Google?
Google's robots.txt guidance says that links from elsewhere can still lead to a blocked URL appearing in results. For index exclusion, noindex is a directive that tells search engines to leave a page out of their index. Webflow's indexing guide documents adding it through Page settings > SEO settings > Sitemap indexing, with the automatically generated sitemap enabled. You switch the toggle off and publish the site.
Can you use this file to protect private content?
Webflow's robots.txt guide states that the file is publicly visible. The same guide warns that some bots ignore its instructions, so private content needs access protection.
Where is the staging subdomain control?
Webflow's indexing documentation places Staging indexing under Site settings > SEO > Indexing. You can switch it off, save, and publish; this produces a separate robots.txt on the webflow.io subdomain. Webflow requires a Site plan or paid Workspace for this control.
Sources
- Webflow, "Set robots.txt rules", 19 May 2026
- Webflow, "Disable search engine indexing", 29 April 2026
- Google Search Central, "Introduction to robots.txt", 10 December 2025
Check how a page reads to answer engines. Open the GEO score analyzer