In short. A User-Agent header presents a client identity, while crawler verification checks the request origin through methods documented by the operator.

Definition: User-agent spoofing is sending a User-Agent HTTP request header that identifies the client as different software. Because the client supplies this value, the header alone cannot prove a bot's identity.

Key facts

As of 17 October 2026, Google's crawler verification documentation describes checking requests from clients claiming to be Google through DNS lookups or published IP address ranges.
The IETF's HTTP Semantics specification explicitly discusses clients masquerading as other user agents and the responses they receive.
Apache's logging documentation describes the logged User-Agent as information the client browser reports about itself.

What does the header actually identify?

A user agent is client software that initiates HTTP requests. In RFC 9110, section 10.1.5, the User-Agent field contains product identifiers, with optional versions and comments. Servers can use this information to adapt responses to software limitations or analyse browser use.

The specification also addresses software presenting another client's identity. It says recipients can interpret that choice as a request for responses tailored to the named software. A product name in the header therefore expresses the identity presented by the sender; independent evidence is needed to attribute a request to a crawler operator. A count based only on matching that name measures requests carrying the name.

How does Google verify a request?

The Domain Name System (DNS) maps names and IP addresses. Google's verification procedure starts with the requesting IP address from server logs. A reverse lookup returns a hostname. After checking the domain, a forward lookup must return the original IP address.

The documentation provides this example:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For automated checks, Google provides address lists for its crawler categories, including common-crawlers.json for common crawlers such as Googlebot. The comparison uses the requesting address.

What should a website measure in its logs?

Apache's Combined Log Format records the User-Agent request header through %{User-agent}i. Its %h field records the remote host, normally an IP address. With HostnameLookups enabled, Apache records the resolved hostname instead.

These fields support separate records of the claimed client identity and the address used for verification. A crawler report can retain the header match alongside the verification result, so its totals distinguish requests carrying a crawler name from requests whose origin passes the operator's checks. This follows from combining Apache's field definitions with Google's verification procedure.

FAQ

Is a reverse DNS lookup sufficient?

Follow Google's complete procedure: check the returned domain and perform the forward lookup. Compare its result with the original requesting IP address.

Can a proxy affect the address in a log?

Yes. Apache's access log explanation says an intervening proxy can make the logged address belong to the proxy. Account for that when selecting the address you verify.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer