How must Cyrillic paths be written in a Yandex robots.txt file?

Definition: Write a Cyrillic page path in the same percent-encoded form used by the site’s URL structure. Write a Cyrillic hostname as its Punycode form. Yandex’s documentation shows the path /корзина as /%D0%BA%D0%BE%D1%80%D0%B7%D0%B8%D0%BD%D0%B0 and the hostname сайт.рф as xn--80aswg.xn--p1ai.

Key facts

• Yandex says that Cyrillic is not allowed in the robots.txt file or in server HTTP headers. Yandex’s robots.txt documentation separates page addresses from domain names for the replacement form.
• A page path uses the encoding already used by the site’s URL structure. The Yandex example maps /корзина to /%D0%BA%D0%BE%D1%80%D0%B7%D0%B8%D0%BD%D0%B0.
• A domain name uses Punycode. The same example maps сайт.рф to xn--80aswg.xn--p1ai.
• RFC 9309 requires octets outside ASCII in a URI or robots.txt path to be percent-encoded before comparison. The Robots Exclusion Protocol also describes the comparison in octets.
• RFC 3986 defines a percent-encoded octet as % followed by two hexadecimal digits. URI Generic Syntax specifies UTF-8 before percent encoding for text from the Universal Character Set.

How should a Cyrillic page path be written?

Percent encoding represents the UTF-8 bytes of characters that cannot appear directly in the URI form. RFC 3986’s percent-encoding definition defines each encoded byte as a three-character sequence consisting of % and two hexadecimal digits. In the Yandex example, the Cyrillic path is therefore written as an ASCII sequence of percent-encoded bytes:

User-agent: Yandex
Disallow: /%D0%BA%D0%BE%D1%80%D0%B7%D0%B8%D0%BD%D0%B0

Yandex states that page addresses use the same encoding as the current site structure. The relevant path is the path component after the hostname, so encode the path in the form the site uses for that URL. RFC 9309 says that non-ASCII octets in the URI and robots.txt paths must be percent-encoded before the crawler compares them. It also says that matching starts at the first octet of the path and should be case sensitive.

Do not apply percent encoding twice. The same RFC says that a URI is in its percent-encoded form once produced and that implementations must not percent-encode or decode the same string more than once. A rule containing %25D0 therefore describes a different sequence from the Yandex example’s %D0 form.

How should a Cyrillic hostname be written?

The hostname is the domain part before the first path slash. Yandex’s robots.txt documentation requires a domain name with Cyrillic characters to use Punycode. Its example is:

User-agent: Yandex
Sitemap: http://xn--80aswg.xn--p1ai/sitemap.xml

Punycode is a transfer encoding for internationalized domain names. RFC 3492 says that it reversibly transforms a Unicode string into an ASCII string, with non-ASCII characters represented by ASCII characters allowed in hostname labels. It also describes ACE labels as ASCII labels with an ACE prefix and a Punycode remainder. In the Yandex example, xn--80aswg and xn--p1ai are the Punycode labels for the two Cyrillic labels.

This host rule applies to the hostname, while the path rule applies after the slash. Keep the two transformations separate when a URL contains both parts. RFC 3986’s URI syntax places the authority before the next slash, and Yandex supplies a distinct example for each component.

What should you check in the file?

Check the actual URL structure and copy its encoded path into the Disallow or Allow value. Compare the complete host in a Sitemap URL with its Punycode form. Yandex’s documentation says its robot considers case in substrings such as a file name or path, while directive names are case insensitive. The page also directs you to create robots.txt, check it in Yandex Webmaster, and place it in the site root.

Related terms

• robots.txt
• UTF-8 encoding
• How should accented and non-Latin characters be encoded in URLs?

FAQ

Can I put Cyrillic directly after Disallow:?

Yandex’s documentation says that Cyrillic is not allowed in the robots.txt file. Use the percent-encoded path form shown in its example.

Should I percent-encode a Cyrillic domain name in the Sitemap URL?

Yandex instructs you to use Punycode for domain names. Its example uses xn--80aswg.xn--p1ai, while the page path is handled separately.

Should I encode the path twice if the URL already contains % sequences?

No. RFC 3986 says that the same string must not be percent-encoded or decoded more than once.

Sources

1. Yandex Webmaster, "Using robots.txt", undated
2. IETF, "RFC 9309: Robots Exclusion Protocol", 2022-09
3. IETF, "RFC 3986: Uniform Resource Identifier (URI): Generic Syntax", 2005-01
4. IETF, "RFC 3492: Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)", 2003-03

---

Check how a page reads to answer engines. Open the GEO score analyzer