In short. Googlebot crawls for Google Search, and its requests can be verified through reverse and forward DNS lookups or published crawler IP ranges.

Definition: Googlebot is Google's search crawler, the software that automatically fetches web pages for Google Search. As of October 2026, Google documents two variants, Googlebot Smartphone and Googlebot Desktop, which simulate mobile and desktop users respectively.

Key facts

Google's common crawlers documentation lists Googlebot as the robots.txt user-agent token and states that common crawlers obey robots.txt rules during automatic crawling.
Google's Googlebot documentation says most sites receive a majority of Googlebot requests from the mobile crawler because Google Search primarily indexes their mobile content.
Google's request verification guide describes manual hostname verification and automatic matching against published Internet Protocol (IP) address ranges.
Google's AI features documentation requires a page to be indexed and eligible for a Search snippet to qualify as a supporting link in AI Overviews or AI Mode.

How does Googlebot identify itself?

A user-agent header is a field in an HTTP request that identifies the requesting software. Google's Googlebot documentation explains that this header distinguishes Smartphone from Desktop. Both variants obey the same robots.txt token, so robots.txt rules apply to both together.

The common crawlers list shows Googlebot/2.1 inside the request headers for both variants. Their longer user-agent strings also contain Chrome/W.X.Y.Z, where the letters represent a changing browser version. Google recommends wildcards for the version when filtering logs. A matching header alone is insufficient proof of identity because another crawler can copy it.

How can a server request be verified?

DNS, the Domain Name System, connects hostnames and IP addresses. A reverse lookup finds a hostname from an address; a forward lookup finds addresses from a hostname. Google's verification guide requires both directions for manual verification.

The process starts with the requesting IP address recorded in the server log. For common crawlers, the guide lists hostname patterns under googlebot.com, including crawl-***-***-***-***.googlebot.com and geo-crawl-***-***-***-***.geo.googlebot.com. A forward lookup of the returned hostname must resolve back to the original requesting address.

Google supplies this example, including the lookup results:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For automated checks across many requests, the same guide links to common-crawlers.json. Its IP ranges use CIDR notation, a format for describing address blocks. The requesting address is checked for membership in those ranges. The file covers common crawlers, so the user-agent header remains relevant for identifying the claimed crawler within that category.

What does Googlebot access change for a website?

A robots.txt file supplies crawl rules; a noindex instruction controls indexing, the inclusion of a page in a search index. Google's Googlebot documentation distinguishes these controls: blocking crawling can still leave a page's URL appearing in search results.

For AI Overviews and AI Mode, Google's AI features documentation identifies robots.txt directives for Googlebot as the control for Search crawling. It lists nosnippet, data-nosnippet, max-snippet and noindex as controls for limiting information shown in Search. Google-Extended addresses training and grounding in some other Google systems.

FAQ

Can I block only Googlebot Desktop in robots.txt?

Both variants use the same token. Google's Googlebot documentation states that robots.txt cannot selectively target Smartphone or Desktop.

Is an IP match specific to Googlebot?

Google's verification guide labels the published file as covering common crawlers such as Googlebot. You can combine IP verification with the request header to identify the claimed crawler.

Does Google-Extended control inclusion in Google Search?

Google's common crawlers documentation states that Google-Extended has no effect on inclusion or ranking in Google Search.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer