How do I record crawler user agents in Apache access logs?

Definition: Apache records a crawler's reported User-Agent request header when the access-log format includes %{User-agent}i. The Combined Log Format includes this header alongside the request and response status.

Key facts

• As of 17 October 2026, Apache HTTP Server 2.4's log file documentation provides a combined format containing both Referer and User-Agent request headers.
• Apache's mod_log_config reference distinguishes defining a format nickname with LogFormat from selecting it for a file with CustomLog.
• RFC 9110, section 10.1.5 defines User-Agent as information about the software originating a request; its value can contain product identifiers and comments.

Which configuration records the header?

Apache's Combined Log Format example supplies these directives:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
CustomLog "log/access_log" combined

LogFormat associates the format string with the nickname combined. CustomLog selects that nickname and the destination file. The relative path log/access_log is resolved against ServerRoot, Apache's server root directory.

The mod_log_config reference permits these directives in the main server configuration or a virtual host, a configuration section for a particular site. Defining the nickname alone does not select a destination or apply the format to a log.

How are the recorded fields read?

The final quoted field in Apache's combined example is the reported user agent. The preceding quoted field is Referer, the page the client reports having come from. The format retains the request line, receipt time and response status.

Apache's format reference specifies %>s for the final response status after an internal redirect. It also explains that other modules can change the request-header value recorded by %{VARNAME}i. For an earlier value, the documented method copies the header to an internal environment variable with mod_setenvif and logs that variable with %{VARNAME}e.

How is the configuration applied and checked?

Apache's apachectl documentation provides a syntax test and a graceful restart command:

apachectl configtest
apachectl graceful

The syntax test reports Syntax Ok or details of an error. After a successful test, the graceful restart applies the configuration while allowing currently open connections to continue. It also performs a configuration check before restarting.

For a site with its own logging directives, Apache's virtual-host logging documentation places that site's requests in its specified file. A virtual host without logging directives sends its requests to the main server logs.

Related terms

• Server log analysis
• User-agent spoofing
• How do I check whether a request really comes from Googlebot?

FAQ

Does a crawler name prove who sent the request?

The header alone does not verify the sender. As of 17 October 2026, Google's request verification documentation describes checking requests through reverse and forward DNS lookups or comparison with published IP ranges. If you need to verify a request claiming to be Googlebot, these methods use its IP address.

Can I keep a separate user-agent log?

Yes. Apache's multiple access logs example uses CustomLog "logs/agent_log" "%{User-agent}i" alongside another access log. You can specify multiple CustomLog directives to write separate files.

Sources

1. Apache Software Foundation, "Log Files", undated
2. Apache Software Foundation, "Apache Module mod_log_config", undated
3. IETF, "RFC 9110: HTTP Semantics", June 2022
4. Apache Software Foundation, "apachectl - Apache HTTP Server Control Interface", undated
5. Google, "Verify Requests from Google Crawlers and Fetchers", updated 20 March 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer