Why does YandexCheckBot visit a site from different regions?

The Citead team, updated

In short. YandexCheckBot visits from different regions to check availability to users, separate from Yandex search indexing.

Definition: YandexCheckBot is a Yandex robot that downloads pages from websites in different regions of the world to check their availability to users. Yandex's robot reference places this activity outside indexing requests.

Key facts

  • Yandex documents this user-agent: Mozilla/5.0 (compatible; YandexCheckBot/3.0; +http://yandex.com/bots). Yandex's robot reference identifies the bot by this string.
  • The reference gives the regional availability check as YandexCheckBot's purpose and places it in the table of robots outside indexing requests.
  • The robot table gives YandexCheckBot No under the general rules specified in robots.txt.
  • Yandex identifies YandexBot/3.0 as the main indexing robot, which gives the two visits different documented purposes.
  • Yandex's availability guidance says a server error can make a site unavailable to visitors as well as prevent the indexing bot from accessing it. Its site accessibility documentation describes both effects.

What is YandexCheckBot checking?

Yandex says that YandexCheckBot downloads pages from websites in different regions of the world to check their availability to users. The regional part is therefore part of the documented test purpose. Yandex does not publish a list of the regions used by this robot, its request schedule, or the response conditions used to evaluate availability.

This visit is separate from a search indexing request. In the same robot table, Yandex describes YandexBot/3.0 as the main indexing robot and YandexCheckBot as a robot whose documented purpose sits outside indexing requests. A YandexCheckBot entry in a server log is evidence of the availability-check function described by Yandex, rather than a documented request to add a page to the search index.

How does availability relate to site access?

Yandex's site accessibility documentation says that its Webmaster tools can check whether pages are accessible to users and indexing bots. It gives a server error as an example that can prevent the indexing bot from accessing a site and make the site unavailable to visitors for the same reason.

The same documentation describes the Server response check tool. It says that a 200 OK response means the page is accessible to indexing bots and site visitors at the moment of checking. Pages with other response codes are described as inaccessible to bots and users and unable to appear in search results. These diagnostics describe access results for the checked page.

How can a site owner identify the request?

Start with the user-agent recorded in the server log:

Mozilla/5.0 (compatible; YandexCheckBot/3.0; +http://yandex.com/bots)

Yandex warns that some robots can pretend to be Yandex robots by sending the relevant user-agent. Its robot verification guidance says to find the request IP in the server log, perform a reverse DNS lookup, and check that the host name ends in yandex.ru, yandex.net, or yandex.com. A forward lookup should then return the IP address from the original request. Yandex also says that bot IP addresses change frequently and that their full list is not disclosed.

Yandex provides a separate IP address check tool. Its results distinguish an IP used by a Yandex indexing bot, an IP belonging to Yandex that may be associated with another Yandex service, and an IP outside Yandex.

Can robots.txt restrict YandexCheckBot?

The robot reference gives YandexCheckBot No under the general robots.txt rules. Yandex's User-agent directive documentation says that some Yandex robots may ignore directives for User-agent: Yandex and that a site can specify the exact user-agent for such robots. A matching rule can be written as:

User-agent: YandexCheckBot
Disallow: /

Yandex's reference also explains that a document downloaded by a special-purpose robot will never be indexed if the main Yandex robot cannot access it. A rule for YandexCheckBot therefore addresses this availability-check request, while indexing access remains a separate question.

Related terms

FAQ

What does YandexCheckBot do for Yandex search?

Yandex describes it as checking page availability from different regions and explicitly places its work outside indexing requests.

Does a YandexCheckBot request establish access for YandexBot?

A separate YandexBot request is relevant. Yandex states that a document downloaded by a special-purpose robot will remain outside search when the main Yandex robot cannot access it.

How can you check whether the request really came from Yandex?

Check the server-log IP with reverse and forward DNS lookups, or use Yandex Webmaster's IP address check tool. The user-agent string alone can be imitated.

Sources

  1. Yandex Webmaster, "How to check that a robot belongs to Yandex", undated
  2. Yandex Webmaster, "How do I check if a site is accessible to users and indexing robots?", undated
  3. Yandex Webmaster, "The User-agent directive", undated
  4. Yandex Webmaster, "Checking if an IP address belongs to Yandex", undated

Check how a page reads to answer engines. Open the GEO score analyzer

Test a page and see what to fix first

The analysis usually takes less than 10 seconds and needs no account.

400,000+ pages scored in the public benchmark