How do I change WordPress's virtual robots.txt output?

Definition: WordPress's virtual robots.txt is a text response generated by WordPress code, separate from a physical robots.txt served directly by the web server. The robots_txt filter passes that response to a callback, a registered function that returns the modified text, before WordPress prints it.

Key facts

• As of 17 October 2026, WordPress's robots_txt reference documents an output string named $output and a boolean $public indicating whether the site is considered public.
• WordPress's do_robots() function builds a wildcard user-agent group, disallows the administration path and allows the admin-ajax.php path before applying the filter.
• The add_filter() reference explains that callbacks return the modified value and execute in priority order.
• WordPress's Apache rewrite generator includes an existing-file check before routing requests to the WordPress index script. A physical robots.txt can therefore bypass WordPress's output filter.

How does the filter change the response?

A filter is a hook that passes a value through registered callback functions. The following illustrative PHP snippet registers a callback in a custom plugin and appends an additional sitemap address while retaining the incoming output. The example address is a placeholder for an existing sitemap.

add_filter( 'robots_txt', function ( $output, $public ) {
    if ( $public ) {
        $output .= "\nSitemap: https://www.example.com/additional-sitemap.xml\n";
    }

    return $output;
}, 10, 2 );

This follows the pattern in WordPress's WP_Sitemaps::add_robots() method: append a Sitemap: line when the public flag is true, then return the output. The custom callback adds a separate sitemap address; core's method supplies the core sitemap index address.

In the registration, 10 is the priority and 2 is the number of arguments accepted. WordPress's filter registration documentation gives the default priority as 10 and the default accepted argument count as 1. Explicitly accepting both arguments makes $public available. Lower priority numbers run earlier, so callbacks running later can further change the response.

What changes when a physical robots.txt exists?

A physical robots.txt is a text file stored on the server. With WordPress's standard Apache routing, an existing file bypasses the fallback route into WordPress. The rewrite generator's source shows the !-f condition that restricts that route to requests without a matching file.

Consequently, a filter callback can be registered correctly while the public URL still serves a physical file. Editing that file changes the server-served content. Changing the virtual response requires the request to reach WordPress's robots handler. Checking which response is served comes before changing PHP callback priorities.

How is the public output checked?

The public response is checked at the site's /robots.txt URL after changing the callback. RFC 9309 specifies this lowercase filename at the service's top-level path, with UTF-8 encoding and the text/plain media type. Its example URL is:

https://www.example.com/robots.txt

For the sitemap example above, the returned text should contain the configured Sitemap: line when $public is true. The check covers the complete response, including rules already present before the callback runs. WordPress's do_robots() source sets a plain-text content type and prints the value returned by apply_filters( 'robots_txt', $output, $public ).

Related terms

• robots.txt
• Where is the search engine visibility setting in WordPress?
• Where is the built-in WordPress XML sitemap?

FAQ

Do I return or print the modified text?

Return the output string from your callback. The robots_txt reference shows that WordPress prints the filtered return value.

Can this filter edit a physical file?

The filter modifies the value generated during WordPress execution. If your server serves a physical robots.txt directly, edit that file; WordPress's standard Apache rewrite rules bypass existing files.

Where does the public flag come from?

WordPress's do_robots() implementation reads the blog_public option and converts it to a boolean before passing it to the filter.

Sources

1. WordPress Developer Resources, "robots_txt", undated
2. WordPress Developer Resources, "do_robots()", undated
3. WordPress Developer Resources, "add_filter()", undated
4. WordPress Developer Resources, "WP_Rewrite::mod_rewrite_rules()", undated
5. WordPress Developer Resources, "WP_Sitemaps::add_robots()", undated
6. IETF, "Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer