In short. Yandex documents YandexSearchShop as a user-requested catalog robot that may ignore general robots.txt rules.
Definition: YandexSearchShop is a Yandex robot that downloads product catalogs in YML files when users request them. Yandex documents that this robot does not follow the general
robots.txtrules used for indexing, so a YML file can be in a directory blocked to the main indexing robot.
Key facts
- Yandex identifies the robot with the user-agent
Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots)in its robot reference. - Its documented purpose is to download product catalogs in YML files by users’ requests.
- Yandex says these files are often placed in directories prohibited for indexing.
- The same table marks
YandexSearchShopas not taking into account the general rules inrobots.txt. - Yandex says some non-indexing robots may ignore restrictive rules written for
User-agent: *.
What does “blocked directory” mean here?
robots.txt is the file that gives search engine robots site indexing parameters. A Disallow rule prohibits crawling of a section or an individual page, according to Yandex’s robots.txt documentation. For example, a site can use this rule for a feed directory:
User-agent: *
Disallow: /feeds/
That rule is a general rule for arbitrary robots. Yandex’s robot reference describes a separate class of robots that download documents for purposes other than indexing. It says those robots may ignore restrictive directives written for User-agent: *, and it lists YandexSearchShop in that class. The exception is about the request made by that robot. It does not turn the catalog into an indexed page.
Why does Yandex use a separate robot?
Yandex describes YandexSearchShop as a catalog retrieval robot rather than the main indexing robot. The reference says its request is triggered by a user request and that YML files are often stored in directories that indexing rules prohibit. This lets a catalog remain outside the ordinary crawl path while still being available to the service that requested it.
The same documentation says that if such a robot downloads a document which the main Yandex robot cannot access, the document will never be indexed and will not be found in search results. A successful YandexSearchShop request therefore describes feed retrieval, not ordinary search indexing.
How can a site owner verify or restrict it?
The request can be identified in server logs by its user-agent:
Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots)
Yandex advises checking a suspected robot with reverse DNS, confirming that its host name ends in yandex.ru, yandex.net or yandex.com, and then using forward DNS to confirm that the address matches. Yandex also says that its robots use changing IP addresses and that their list is not disclosed.
When the general wildcard rule is too broad for a catalog, Yandex’s User-agent directive documentation says that a site can give a robot its own group. It also says that some Yandex robots can ignore even a User-agent: Yandex group, so the exact user-agent should be specified for those robots. A matching restriction can therefore be written as:
User-agent: YandexSearchShop
Disallow: /feeds/
Yandex’s robot reference gives this robot-specific approach for restricting non-indexing robots. The reference does not document a separate YML authentication token or a published fixed IP list for YandexSearchShop.
Related terms
FAQ
Does a YandexSearchShop visit index the YML file?
No. Yandex describes it as a catalog download made by a user’s request. Its documentation says that a document inaccessible to the main Yandex robot will not be indexed or found in search results.
Does Disallow: /feeds/ always stop YandexSearchShop?
The general User-agent: * group may be ignored by this robot. To restrict it, specify User-agent: YandexSearchShop and the relevant Disallow path.
How can I tell whether the request came from Yandex?
Check the user-agent in the server log, then follow Yandex’s reverse DNS and forward DNS checks. A matching Yandex host name and address provide the checks described in its robot documentation.
Sources
- Yandex Webmaster, "How to check that a robot belongs to Yandex", undated
- Yandex Webmaster, "Using robots.txt", undated
- Yandex Webmaster, "The User-agent directive", undated
Check how a page reads to answer engines. Open the GEO score analyzer