How do I record crawler request duration in Apache logs?

Definition: Request duration in an Apache access log is the time Apache takes to serve a request. A custom log format can record this duration with %D in microseconds, alongside the request and its User-Agent header for crawler analysis.

Key facts

• As of October 2026, Apache's Log Files guide documents LogFormat for selecting log contents and CustomLog for choosing the destination and format.
• The mod_log_config reference defines %D in microseconds, %T in seconds and %{ms}T in milliseconds.
• Apache's apachectl reference provides configtest to check configuration syntax and graceful to restart while preserving open connections.

Which custom log format records duration?

A log format is a string whose percent directives are replaced by request values. Apache's Log Files guide shows the Combined Log Format, which includes the request line, response status, response size, Referer and User-Agent headers. This adapted configuration appends a labeled duration field:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" duration_us=%D" combined_timed
CustomLog "logs/access_timed_log" combined_timed

combined_timed is the format nickname used by CustomLog. The relative filename resolves beneath ServerRoot, Apache's server root directory. Literal text such as duration_us= appears in the output, and the escaped quotes preserve quoted request and header fields.

The same guide describes User-Agent as identifying information reported by the client. The example records all requests reaching this log; crawler analysis can select records by that header afterward. Apache documents multiple CustomLog directives for writing multiple access logs, so adding this destination alongside an existing directive creates an additional log.

Which timing field should the format use?

The timing field reference specifies %{UNIT}T, with ms, us and s selecting milliseconds, microseconds and seconds. Unit-qualified %T is available in Apache HTTP Server 2.4.13 and later. %{us}T gives the same result as %D; %{s}T gives the same result as %T.

For milliseconds, the duration portion of the example becomes:

duration_ms=%{ms}T

The field label changes with the selected unit. The reference also states that %D and %T consult the original request by default when internal redirects occur; %>s selects the final response status.

How is the configuration applied and checked?

The apachectl manual documents this syntax test:

apachectl configtest

It reports Syntax Ok or details of a syntax error. After a successful check, the documented graceful restart command is:

apachectl graceful

This command also checks configuration syntax before restarting. Existing connections remain open, and old log files can remain open temporarily.

For websites configured as virtual hosts, meaning separate website configurations within a server, Apache's virtual-host logging guidance states that a CustomLog inside a <VirtualHost> section sends that host's requests to the specified file. Checking a new request therefore includes checking the log destination configured for the relevant host.

Related terms

• Server log analysis
• How do I record crawler user agents in Apache access logs?
• How do I verify an OpenAI crawler request?

FAQ

Does defining a format nickname start writing the log?

The LogFormat documentation states that a nickname definition only defines the nickname. You must reference it in CustomLog to use it for the destination shown above.

Does the existing timestamp already record duration?

Apache's Log Files guide defines %t as the time the request was received. To record elapsed serving time alongside it, you can add the duration field shown above.

Sources

1. Apache Software Foundation, "Log Files", undated
2. Apache Software Foundation, "Apache Module mod_log_config", undated
3. Apache Software Foundation, "apachectl - Apache HTTP Server Control Interface", undated

---

Check how a page reads to answer engines. Open the GEO score analyzer