How can page content be excluded specifically from Yandex AI answers?

Definition: Yandex documents YandexAdditionalBot as the robots.txt user agent for preventing page content from appearing in Search with Yandex AI responses. Add a Disallow rule in a group addressed to that user agent, then wait for Yandex robots to recrawl the page.

Key facts

• Yandex AI answers are generated from pages that Yandex has indexed, according to Yandex Webmaster's Search with Yandex AI guidance.
• The same guidance says to add a Disallow directive for YandexAdditional or YandexAdditionalBot when site information should not be used in Yandex AI answers.
• Yandex's robot table lists YandexAdditionalBot and YandexAdditional as user-agent forms that help process robots.txt for Search with Yandex AI responses. The table says they apply to pages indexed by the primary crawler and do not make indexing requests.
• Yandex says the change takes effect when its robots recrawl the page, typically in 2 to 14 days.
• Yandex's robots.txt documentation says Disallow prohibits crawling of site sections or individual pages, while pages restricted in robots.txt can still participate in Yandex search.

What does the Yandex AI rule control?

Yandex AI is a search feature that provides a concise answer to a query. Yandex says it scans the internet for relevant information, analyzes the sources in real time, and combines the findings into an answer with references. Its webmaster guidance states that these answers are generated from pages already indexed by Yandex.

The separate YandexAdditionalBot rule therefore targets the use of an indexed page in Yandex AI answers. Yandex's robot list describes the bot as helping process robots.txt to prevent page content from appearing in Search with Yandex AI responses. The same row says that the bot applies to pages indexed by the primary crawler and does not make indexing requests. A robots.txt rule for this user agent is consequently an AI-answer control, not a request to remove the URL from Yandex's ordinary index.

Which robots.txt rule should you add?

For one page, Yandex's documented example is:

User-agent: YandexAdditionalBot
Disallow: /cats/mycat.html # prevents the content of the page from appearing in Yandex AI answers

Replace the path with the URL path of the page to exclude. Yandex's robots.txt documentation defines Disallow as a prohibition on crawling sections or individual pages. Its robot documentation also says that the robot name and path are case-sensitive for substrings, while directive names are case-insensitive. Keep the group addressed to YandexAdditionalBot when the rule is intended for that bot specifically.

Yandex lists YandexAdditional as an alternative user-agent form for the same AI-response processing function. Its User-agent directive documentation says that directives for a specific robot take precedence over User-agent: *, and that some Yandex robots may ignore a generic User-agent: Yandex group. A specific group is therefore the relevant form for this opt-out.

How should you check the change?

Yandex requires the file to be named robots.txt, placed in the site's root directory, and accessible to indexing bots. The server should return HTTP 200 OK, or redirect to another robots.txt file that returns HTTP 200 OK. The file size must not exceed 500 KB. Yandex's documentation instructs site owners to check the file in Yandex Webmaster before placing it at the site root.

After publishing the rule, allow for the documented 2 to 14 day recrawl period. For a request seen in server logs, Yandex publishes the full form Mozilla/5.0 (compatible; YandexAdditionalBot/1.0; +http://yandex.com/bots) and also the YandexAdditional/1.0 form. Yandex recommends reverse DNS checking when a request's identity matters: the reverse host should end in yandex.ru, yandex.net, or yandex.com, and a forward lookup should return the original IP address.

Related terms

• YandexBot
• Yandex Alice AI Search
• Yandex Neuro
• robots.txt

FAQ

Does this rule remove a page from Yandex search?

No. Yandex says pages restricted in robots.txt can participate in Yandex search. The YandexAdditionalBot row also describes pages that have already been indexed by the primary crawler.

Can you exclude one URL instead of the whole site?

Yes. Add the URL path after Disallow, as in Yandex's /cats/mycat.html example. The robots.txt documentation describes the directive as covering individual pages or sections.

How long should you wait after publishing the rule?

Yandex says the change takes effect when its robots recrawl the page, typically in 2 to 14 days.

Where can you test the file?

Use the robots.txt checker in Yandex Webmaster before placing the file at your site's root. You can also inspect server logs for the documented YandexAdditionalBot or YandexAdditional user-agent forms.

Sources

1. Yandex Webmaster, "Search with Yandex AI", undated
2. Yandex Webmaster, "Using robots.txt", undated
3. Yandex Webmaster, "The User-agent directive", undated
4. Yandex Webmaster, "How to check that a robot belongs to Yandex", undated

---

Check how a page reads to answer engines. Open the GEO score analyzer