In short. A bot name in a server log identifies a claim that needs source-address verification using the operator's documented method.
Definition: A bot name in a server log is a claimed identity when it comes from the User-Agent HTTP request header, a field in which the client describes itself. Header spoofing means sending that field with another client's identifying text; operator verification checks the request's source address against the operator's documented evidence.
Key facts
- As of 17 October 2026, Google's request verification documentation describes DNS checks and matching source addresses against published IP ranges.
- Google's Googlebot documentation warns that other crawlers often spoof Googlebot's user-agent header and recommends verification before blocking it.
- Apache's Log Files documentation describes the logged User-Agent value as information the client reports about itself.
Why can a crawler name be misleading?
An access log records requests processed by a web server. In Apache's Combined Log Format, the User-Agent field comes from an HTTP request header. Recording that value preserves the client's description of itself.
Google's Googlebot guidance explicitly warns about crawlers borrowing its identifying header. A log filter that searches for Googlebot therefore groups requests carrying that name, including impersonators. Treating the resulting count as verified Google traffic skips the source-address check Google recommends.
For reporting, a useful convention is to retain the claimed name and record the verification result separately. This follows Google's distinction between a request claiming to come from Google and a request whose origin has been checked.
How does Google's verification work?
Google's verification procedure uses the domain name system, or DNS, to translate between addresses and names. A reverse lookup starts with the accessing IP address; a forward lookup translates the returned hostname back into an address.
The documented check examines the returned domain and confirms that the forward lookup returns the original address. Google's common-crawler hostname patterns end in googlebot.com. Its example is:
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
These are documentation example values. An actual check starts with the address in the request log.
For automated checks, the same documentation provides separate address lists for crawler categories. It associates common crawlers such as Googlebot with common-crawlers.json.
Which address should be checked behind a proxy?
A proxy is an intermediary between the requesting client and the server. Apache's log documentation warns that the remote-host address in its example can identify this intermediary instead of the originating machine.
That detail changes the verification input: checking a proxy's address checks the intermediary. Before applying the operator's method, the log review needs to establish which address field represents the originating request. Keeping the original log record alongside the verification result makes that choice reviewable.
Related terms
FAQ
Can you identify Googlebot from its name alone?
No. Google's Googlebot documentation warns that other crawlers spoof its header. You can retain the name as a claimed identity while checking the request's source.
Can you stop after the reverse DNS lookup?
Google's procedure also requires a forward lookup and a match with the original address. You need that confirmation to complete its DNS check.
Sources
- Google, "Verify requests from Google crawlers and fetchers", 20 March 2026
- Google Search Central, "Googlebot", 3 February 2026
- Apache Software Foundation, "Log Files", accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer