How can a publisher ask Brave to re-fetch a newly noindexed page?

Definition: A publisher can add a noindex directive to a page, then ask Brave to re-fetch that URL through Brave's URL submission page. Brave says the re-fetch is needed before it can apply the directive and delist the page.

Key facts

• Brave says its crawler discovers pages and indexes their content. Its Brave Search Crawler help page also says that a page or domain outside Googlebot's crawlable scope is outside Brave's crawl as well.
• Brave says robots.txt is not the mechanism it uses to prevent a page from being indexed. It names the robots noindex directive as the page-level delisting control.
• After the directive is added, Brave says it needs to re-fetch the page to apply the change. The same help page links to Brave's URL submission page for that request.
• Google's noindex documentation describes two implementation forms, a meta tag and an HTTP response header, with the same effect for Google.
• Google says a crawler must be able to fetch the page or resource to see its tags and HTTP headers. A robots.txt block prevents that crawler from reading the directive.

What is the Brave re-fetch workflow?

First publish the page change. For an HTML page, add a robots noindex directive. Then open Brave's Submit URL page, enter the page URL in the field labeled “Insert the URL to be re-fetched”, and select Submit. Brave's crawler help page identifies this page as the route for a re-fetch request.

The request concerns one URL. Brave's wording connects the sequence directly: the page is updated with noindex, Brave re-fetches it, and the re-fetch applies the change by delisting the page. The help page does not publish a completion interval for this process.

How should the page expose noindex?

Google documents two ways to send the directive. An HTML page can place the meta tag in its head section:

<meta name="robots" content="noindex">

For a resource such as a PDF, video file, or image file, Google documents the HTTP response form:

X-Robots-Tag: noindex

Google says the two forms have the same effect for its crawler and that noindex in robots.txt is unsupported. Brave's help page names the robots noindex directive without documenting a separate Brave-specific header syntax, so the implementation should match the resource type and the publisher's server or CMS setup.

What must be crawlable before the request?

The page must remain fetchable while Brave processes the request. Brave says that its bot does not crawl a domain or page that is not crawlable by Googlebot. Google likewise says the page or resource must be accessible to the crawler and must not be blocked by robots.txt, because a blocked crawler cannot read the noindex rule.

That makes the order important: publish the page-level directive, keep the URL available for crawling, and then submit the URL through Brave's form. A robots.txt rule alone does not supply the page-level directive described by Brave's crawler help.

Related terms

• noindex
• Brave Search API
• Does Brave publish a separate search crawler user agent?
• robots.txt Disallow directive

FAQ

Does robots.txt remove the page from Brave's index?

Brave says robots.txt is not used to prevent a page from being indexed. Its documented route is a page-level robots noindex directive followed by a re-fetch request.

Can a publisher submit a URL that no longer exists?

Yes. Brave's crawler help page says a publisher can submit the URL of a page that no longer exists to request delisting. It also gives not-found@brave.com as a contact address for that case.

What if the URL contains personal data?

Brave directs inquiries about personal data to its Right To Be Forgotten guidance and process. That route is separate from the publisher's ordinary noindex re-fetch workflow.

Sources

1. Brave Search, "Brave Search Crawler", 14 August 2020
2. Brave Search, "Submit URL - Brave Search", undated
3. Google Search Central, "Block Search Indexing with noindex", 10 December 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer