In short. Google-CloudVertexBot handles site-owner-requested Vertex AI crawling and follows robots.txt rules during automatic crawls.

Definition: Google-CloudVertexBot is Google's crawler for website crawls requested by site owners to build Vertex AI Agents. As of 17 October 2026, Google lists it among its common crawlers, which obey robots.txt rules during automatic crawling.

Key facts

Google's common crawlers documentation identifies Google-CloudVertexBot as both an HTTP user-agent substring and a robots.txt token; it also lists Googlebot as an applicable token.
Google Cloud's data preparation guide requires access for Google-CloudVertexBot to crawl and index website content, including paywalled content.
Google's robots.txt specification applies the most specific matching user-agent group and keeps specific groups separate from the global * group.
Google's request verification guide describes checking a request's source IP address through published address ranges or reverse and forward DNS lookups.

What does the site owner request?

Google Cloud's website data instructions describe creating a data store by supplying web page URLs for Google to crawl and index for searching or recommending. A data store holds the website data used by the application. Its configuration includes URL patterns to include and exclude.

The guide requires domain verification for advanced website indexing. It also tells site owners to exclude dynamic search URLs, giving example.com/search?* as an example: indexing each unique search URL increases index storage costs and dilutes search quality.

Google's common crawlers list assigns Google-CloudVertexBot to these owner-requested Vertex AI crawls. It states that crawling preferences addressed to this user agent have no effect on Google Search or other products.

How does robots.txt control access?

A robots.txt file gives crawlers rules for accessing website paths. Google Cloud's preparation guide provides this group to allow Google-CloudVertexBot access:

user-agent: Google-CloudVertexBot
allow: /

Google's robots.txt specification requires the file at the site's top-level directory. The user-agent field selects the crawler, and allow: / permits crawling from the website root. A disallow rule specifies paths the selected crawler must avoid. When a specific group applies, rules in the global * group are not combined with it.

Sitemaps need a separate access check. The same Cloud guide says Agent Search crawls website pages with Google-CloudVertexBot but fetches the sitemap file with Googlebot. If robots.txt, a content delivery network, or a firewall blocks Googlebot, the sitemap's URLs cannot be indexed or refreshed through that sitemap even when page crawling remains possible.

How can a website check crawler requests?

An HTTP user-agent string identifies the client making a request. Google's common crawlers list gives Google-CloudVertexBot as the substring to look for and warns that user-agent strings can be spoofed.

Google's verification instructions use the source IP address recorded in server logs. For a manual check, a reverse DNS lookup finds its hostname; a forward lookup must then resolve that hostname to the original IP address. For common crawlers, the documented hostname patterns end in googlebot.com or geo.googlebot.com. Automated checks can compare source addresses against Google's published common-crawlers.json ranges.

FAQ

Does blocking Google-CloudVertexBot affect Google Search?

Google's crawler documentation states that preferences addressed to Google-CloudVertexBot have no effect on Google Search or other products. Its documented scope is site-owner-requested crawling for Vertex AI Agents.

Why does my sitemap need Googlebot access?

Google Cloud's preparation guide assigns sitemap fetching to Googlebot. If you use a sitemap to index or refresh website pages, allow Googlebot to fetch that file as well as allowing Google-CloudVertexBot to crawl the pages.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer