In short. Use RobotsTxt on each Drupal multisite site, remove the static root file, exclude it from Composer scaffolding and check each hostname’s /robots.txt response.
Definition: Drupal multisite runs multiple websites from a single codebase. As of October 2026, the RobotsTxt module generates robots.txt dynamically and allows its contents to be edited separately for each site through Drupal's web interface.
Key facts
- The RobotsTxt README requires the physical robots.txt in the Drupal installation root to be removed or renamed so the module can handle requests.
- The module's routing configuration exposes
/robots.txtpublicly and protects its editing form with theadminister robots.txtpermission. - Google's robots.txt specification limits a file's rules to the host, protocol and port where it is served.
How is the content configured for each site?
The module README places its editor under Administration > Configuration > Search and metadata > RobotsTxt. The workflow is to install and enable the module for the sites concerned, open each site's administration interface, edit that site's robots.txt content and save it.
The routing file defines the editor at this path:
/admin/config/search/robotstxt
Per-site editing is the module's documented multisite feature. Each site's saved content therefore needs a separate review when its intended crawling rules differ from those of the other sites sharing the codebase.
Why must the physical root file be removed?
The README's installation instructions explain that the web server serves an existing static robots.txt before the module receives the request. Renaming or removing that file allows the generated response to reach the public endpoint.
Because the installation shares a codebase, this change needs to be coordinated across the sites using that root file. The implementation review should cover module enablement and saved rules for every affected site before the shared static file is retired.
How is Composer prevented from restoring the static file?
For projects using Drupal Scaffolding, the RobotsTxt README documents a file-mapping exclusion for robots.txt. The relevant mapping inside the existing drupal-scaffold configuration is:
"file-mapping": {
"[web-root]/robots.txt": false
}
This setting tells Composer to ignore that scaffold file. It belongs alongside the project's existing scaffold mappings, so their other settings remain present.
How is the result checked for every site?
The module's public route is:
/robots.txt
Opening that path on each site's hostname and comparing the returned text with its saved configuration checks the served result. Google's specification gives https://example.com/robots.txt as the general case and rejects robots.txt files in subdirectories. Different subdirectory paths on one hostname therefore do not provide independent robots.txt endpoints.
If the saved content fails to appear, the module README's troubleshooting checklist covers a remaining static file, disabled clean URLs and the Fast 404 feature.
Related terms
FAQ
Where do I find each generated file on disk?
The README describes dynamic generation and per-site web editing. You check the result at the site's public /robots.txt endpoint.
Does a robots.txt block make my development site private?
Google's introduction to robots.txt recommends password protection for private files because robots.txt instructions rely on crawler compliance. You need access controls to protect private content.
Sources
- Drupal RobotsTxt project, "RobotsTxt README", undated
- Drupal RobotsTxt project, "robotstxt.routing.yml", undated
- Google Search Central, "How Google interprets the robots.txt specification", 31 August 2026
- Google Search Central, "Introduction to robots.txt", 10 December 2025
Check how a page reads to answer engines. Open the GEO score analyzer