Why can AIOSEO's crawler blocking fail with a static robots.txt file?

Definition: AIOSEO's Unwanted Bots setting adds selected crawler rules to WordPress's dynamic robots.txt output. A static robots.txt file is a separate file that AIOSEO does not modify, so the setting can appear enabled while the response at /robots.txt lacks those rules.

Key facts

• AIOSEO's Crawl Cleanup documentation says the Unwanted Bots tab controls which bots are blocked from crawling the site.
• The setting lets you select individual crawlers or All AI Crawlers, which applies to all crawlers listed in that interface.
• When a box under Block AI Crawlers is selected, AIOSEO adds rules to the dynamic robots.txt that WordPress creates.
• A static robots.txt file is outside that output path. AIOSEO says it will not write to the static file and recommends deleting it and using WordPress's dynamic output.
• WordPress exposes a robots_txt filter in its do_robots() implementation, which gives plugins a place to alter generated output.

What does AIOSEO's Unwanted Bots setting change?

As of August 14, 2026, AIOSEO's documentation says that Unwanted Bots controls which bots are blocked from crawling the site. It also says that selecting a crawler under Block AI Crawlers makes All in One SEO add rules to the dynamic robots.txt created by WordPress.

The important distinction is where the rule is written. The dynamic response is produced through WordPress's robots endpoint, so AIOSEO can add its rules while WordPress builds that response. A static file is already separate from that generated output. The same AIOSEO documentation says that the plugin will not write to a static robots.txt file and recommends deleting the file so the WordPress version is used.

That creates a clear diagnostic pattern: the setting can be enabled in WordPress, while a request for /robots.txt still returns a static response without the selected crawler group. The browser or crawler follows the response it receives at that path, so the generated rules are absent from the artifact being evaluated.

How does WordPress provide the output AIOSEO uses?

WordPress's do_robots() reference describes the function as displaying the default robots.txt file content. Its source sends a plain text response and applies the robots_txt filter to the output.

apply_filters( 'robots_txt', $output, $public );

This is the integration point relevant to the AIOSEO setting. A plugin can change the generated string when WordPress handles the robots request. A file that is served independently of this generated response does not pass through that filter, which is why the AIOSEO rule can be missing even though the setting is saved.

How can you check which rules a crawler receives?

Start with the exact root resource required by the Robots Exclusion Protocol: /robots.txt. RFC 9309 says that the rules must be accessible in a file named /robots.txt in the top-level path of the service. Request that URL on the site's public hostname and inspect the returned text.

The protocol organizes rules into groups. A group starts with a user-agent line and continues with rules for that product token. Its example uses this form:

user-agent: ExampleBot
disallow: /foo

RFC 9309 says crawlers match the product token case insensitively and obey the matching group's rules. If no specific group matches, a user-agent: * group applies when present. Therefore, compare the live response with the crawler selections in AIOSEO: look for the relevant user-agent group and its allow or disallow rules, then check whether the response is the static file or WordPress's generated output. If the selected rules are missing, remove the static file as AIOSEO recommends, then request /robots.txt again.

Related terms

• robots.txt
• How do I change WordPress's virtual robots.txt output?
• robots.txt User-agent directive

FAQ

Does selecting All AI Crawlers create a physical file?

No physical file creation is documented here. AIOSEO says the selection adds rules to the dynamic robots.txt created by WordPress.

What should you do when a static file exists?

AIOSEO recommends deleting the static robots.txt file and using the dynamic robots output provided by WordPress. Check the public /robots.txt response after the change.

What should you compare during a test?

Compare the crawler group and its rules in the public /robots.txt response with the selections in the Unwanted Bots tab. RFC 9309 defines the product-token matching and group rules that determine which directives apply.

Sources

1. AIOSEO, "How to Use Crawl Cleanup to Increase Search Engine Crawl Quota", August 14, 2026
2. WordPress Developer Resources, "do_robots()", n.d.
3. IETF, "Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer