Why can YandexRenderResourcesBot fetch disallowed JavaScript resources?

Definition: YandexRenderResourcesBot is a Yandex robot that loads resources needed to render a page with JavaScript. Yandex documents a conditional exception: it ignores a robots.txt rule on a resource when the HTML page that uses the resource is accessible to the Yandex bot.

Key facts

• Yandex identifies Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) as the robot that loads resources for page rendering with JavaScript. Yandex's robot reference lists its robots.txt setting as No.
• The exception is conditional on the containing HTML page being accessible to the Yandex bot. The resource URL can match a Disallow rule while the page that references it remains crawlable. Yandex's robot reference states this directly.
• Yandex says the resource bot does not access resources when the HTML pages that use them are disallowed in robots.txt. The same robot reference sets the boundary.
• Yandex defines Disallow as a directive that prohibits crawling sections or individual pages of a site. Its robots.txt documentation describes the directive's scope.

What is Yandex checking?

The relevant object is the containing HTML page. Yandex's table describes YandexRenderResourcesBot as a renderer resource loader, then says it ignores robots.txt instructions when the HTML page where those resources are located is accessible to the Yandex bot. That makes the page's crawlability the first condition. A JavaScript file, stylesheet, image, or another requested asset can therefore have a path that matches Disallow while the HTML document that calls it is open to Yandex.

The second condition reverses the result. Yandex's documentation says the bot does not access resources if the HTML pages where those resources are used are disallowed in robots.txt. The rule is about the page that contains or uses the resource, rather than an unrestricted promise to fetch every blocked asset.

How does robots.txt affect the two URLs?

Yandex describes robots.txt as a text file containing site indexing parameters for search engine robots. It defines Disallow as prohibiting crawling of sections or individual pages, and lists User-agent as the directive that selects which robot a rule applies to. A resource rule and a document rule can therefore match different URL paths.

Yandex also says its robot takes the case of substrings into account for a file name, path, or robot name, while directive names are case-insensitive. A path review should use the exact URL path and the applicable User-agent group.

User-agent: * #indicates which bots the directives are set for
Disallow: /bin/ # prohibits links from the “Shopping Cart”.
Disallow: /search/ # prohibits links from the site’s built-in search

How can you check the decision?

Start with the HTML URL, then inspect the resource URL separately. Yandex's Robots.txt analysis tool accepts a complete URL or a path relative to the site root in its URL list. The tool shows an allowed icon when a URL can be indexed by Yandex bots and highlights the address in red when it cannot.

For rendered content, Yandex Webmaster provides a Check page workflow and a JavaScript page rendering tool. Yandex says the latter can show how pages appear in a browser and which content is accessible with and without JavaScript execution. Compare the HTML page's result with the resource path's result, and record the user agent and response status from server logs for each request.

Related terms

• YandexBot
• robots.txt
• Rendered DOM

FAQ

Does a Disallow rule on a JavaScript file always stop YandexRenderResourcesBot?

No. When the HTML page using the file is accessible to the Yandex bot, Yandex documents that YandexRenderResourcesBot ignores the resource rule.

What happens when the HTML page is disallowed?

Yandex states that the resource bot does not access resources used by HTML pages disallowed in robots.txt.

Where can a site owner test a URL?

Use Yandex Webmaster's Robots.txt analysis tool. Its URL list accepts a complete URL or a path relative to the site root, then marks the result as allowed or highlights it in red.

Sources

1. Yandex Webmaster, "How to check that a robot belongs to Yandex", undated
2. Yandex Webmaster, "Using robots.txt", undated
3. Yandex Webmaster, "Robots.txt analysis", undated
4. Yandex Webmaster, "How do my site pages appear to indexing bots and users, and how can I influence that?", undated

---

Check how a page reads to answer engines. Open the GEO score analyzer