Does a physical robots.txt file override Adobe Commerce's generated rules?

Definition: Yes. Adobe Commerce uses its generated rules when a request for robots.txt reaches Commerce instead of a physical file. A physical /robots.txt response therefore takes precedence over the rules produced by the Commerce robots controller for that request.

Key facts

• Adobe's SEO overview, updated on October 8, 2026, says Commerce can generate and manage crawler instructions and dynamically route robots.txt to its controller when the request reaches Commerce rather than a physical file. Adobe's SEO overview
• The Magento robots router matches the request path robots.txt, then resolves the index/index action in the robots module. Magento's Router.php
• The frontend route maps the robots front name to Magento_Robots. Magento's routes.xml
• The controller returns a page with the response type text/plain. Magento's Index.php

What does precedence mean in this request flow?

Adobe describes the boundary directly in its Search engine optimization overview. Commerce has a configuration for generating crawler instructions, but the application handles the request only when it reaches Commerce. When a physical file supplies /robots.txt, that response is outside the generated controller output. Editing Commerce's Search Engine Robots settings therefore changes the application path, while a physical file continues to supply its own content when that is the path used for the request.

The distinction concerns the response path, not whether the text happens to contain similar directives. A physical file can contain the same User-agent and Disallow fields as Commerce's output and still be a separate source of the response.

How does the Commerce fallback match /robots.txt?

The official Magento Router.php source describes a router that matches an application action when robots.txt is requested. Its match() method trims the request path, returns no action for another identifier, looks up the robots front name, and creates the index/index action from the first matching module.

$identifier = trim($request->getPathInfo(), '/');
if ($identifier !== 'robots.txt') {
    return null;
}

The frontend route maps the robots front name to Magento_Robots:

<route id="robots" frontName="robots">
    <module name="Magento_Robots"/>
</route>

The frontend route is the application mapping used by the Magento robots module. This is the application fallback described by Adobe when a physical file is absent from the response path.

What does the generated response contain?

Adobe's configuration instructions use Content > Design > Configuration, then the Search Engine Robots section. The page offers a Default Robots setting, a custom instruction field, and Reset to Default. Adobe's default example includes paths such as /index.php/, /checkout/, and /app/:

User-agent: *
Disallow: /index.php/
Disallow: /checkout/
Disallow: /app/

The generated output is assembled by the controller's robots_index_index page handle, and the controller sets Content-Type to text/plain. The configured directives apply to this generated application response. They are not a mechanism for changing a separate physical file.

How can you check which path answered?

Request the exact root path and inspect the body and headers:

curl -i https://www.example.com/robots.txt

Compare the returned text with the physical file deployed at the web root and with the Search Engine Robots configuration in Commerce. The controller's documented response type is text/plain, while the decisive evidence is the content that the request receives. If the physical file is the content served for /robots.txt, Commerce's generated rules are not the rules being returned for that request. If the request reaches the application, the router and controller described above produce the generated response.

Related terms

• robots.txt
• Where does a robots.txt file need to be hosted?
• What happens when robots.txt returns a server error?

FAQ

If I edit Search Engine Robots, will a physical file change?

No. The setting controls the rules Commerce generates for the application response. A physical /robots.txt has its own content and remains the response source when that file is served.

What path does the Commerce router accept?

The router trims the request path and accepts the identifier robots.txt. It returns no matched action for another identifier, then resolves the index/index action for the robots module.

What response type does the controller set?

The controller sets Content-Type to text/plain after creating the page and adding the robots_index_index handle.

Sources

1. Adobe Commerce, "Search engine optimization", 8 October 2026
2. Magento Open Source, "Router.php", undated
3. Magento Open Source, "routes.xml", undated
4. Magento Open Source, "Index.php", undated

---

Check how a page reads to answer engines. Open the GEO score analyzer