In short. A custom Duda robots.txt changes crawler directives, while Duda security systems can block requests before those directives are read.
Definition: No. A custom Duda robots.txt file changes crawler directives, while Duda's security systems can reject requests before robots.txt is read. A 403 response therefore needs platform review even when the uploaded file permits the crawler.
Key facts
- Duda publishes default
sitemap.xml,robots.txt, andllms.txtfiles, and its site configuration documentation describes uploading custom files to the website root. - The Robots Exclusion Protocol defines robots.txt rules as crawler instructions that are requested to be honored, and states that they are not access authorization.
- Duda's traffic and security documentation says unusual automated traffic can produce a 403 error or a blocked page and that request checks happen before robots.txt is read.
What does a custom Duda robots.txt file control?
Duda's documentation says that the default files are available on published sites and cannot be edited directly. A custom file can replace the default robots file when the URL Redirect entry uses robots.txt as the exact Source URL. The documented supported types include .xml, .txt, and .json, and Duda says that a .txt upload can be any text file.
The upload is a file response at the domain root, which matches the standard location for robots.txt:
https://www.example.com/robots.txt
In Duda's workflow, you open More, Settings, and URL Redirect, add a redirect, choose the File option, upload the file, set the type to 2xx OK (Redirect), and publish or republish the site. Duda also says it does not validate custom files, so a corrupt upload can still be served.
Why does a 403 remain after the file is changed?
The IETF specification describes robots.txt as a set of groups and rules for crawler access. It says crawlers are requested to honor these rules and separates them from access authorization. A crawler can therefore receive a response from a platform security layer before it downloads or parses the file.
Duda documents this order for its own platform. Its traffic article says that unusual automated traffic can include third-party crawlers and scanners, that a blocked tool usually reports a 403 error or a blocked page, and that Duda security systems check requests before robots.txt is read. The article says changing robots.txt does not unblock the tool. It also documents that there is no setting to allow a specific IP address.
How should a blocked crawler be investigated?
Check the live response for the crawler's request and record the URL, date, time, response status, user agent, and IP address. Duda asks for the tool's name and what it does, its IP addresses and user agent, and the URL, date, and time of a recent blocked request when support reviews a tool. The documentation says that not every tool can be allowed and that the support team reviews the details.
If expected traffic exceeds 1000 requests per minute, Duda recommends contacting support a week in advance and providing anticipated peak traffic dates and estimated concurrent hits. This is a traffic planning step, separate from editing the crawler file.
Related terms
- robots.txt
- robots.txt User-agent directive
- robots.txt Disallow directive
- What happens when robots.txt returns a server error?
FAQ
Can a permissive robots.txt allow a crawler that Duda blocks?
No. Duda documents that its security systems check requests before robots.txt is read, so a permissive crawler group does not remove that platform block.
What should you send Duda support?
Send the tool name and purpose, its IP addresses and user agent, and the URL, date, and time of a recent blocked request. Duda may ask for more details during its review.
How do you restore Duda's generated robots.txt?
Delete the custom file from URL Redirect settings, then publish or republish the site. Duda documents that the generated file takes effect after that publish step.
Sources
- Duda Support, "Site Configuration Files", 15 October 2026
- Duda Support, "How Much Traffic Can Duda Sites Handle?", 15 October 2026
- Internet Engineering Task Force, "Robots Exclusion Protocol", September 2022
Check how a page reads to answer engines. Open the GEO score analyzer