How do I measure bandwidth used by AI crawlers?

Definition: An AI crawler bandwidth report sums the bytes transferred for requests attributed to verified AI crawlers during a defined period. It identifies the measurement point and byte field so that the total has a clear meaning.

Key facts

• As of 17 October 2026, Apache's access log documentation describes CustomLog for selecting the log destination and LogFormat for selecting recorded fields.
• Apache's log format reference distinguishes response size (%B) from actual network bytes sent (%O).
• Apache's mod_logio documentation defines %I as received bytes and %O as sent bytes, including headers.
• On the same date, OpenAI's crawler documentation lists separate published address feeds for OAI-SearchBot and GPTBot.

Which log fields are needed?

An access log records requests processed by a server. Apache's log format reference documents the fields used in this adapted configuration. It requires mod_logio to be enabled:

LogFormat "%v %a %{c}a %t \"%r\" %>s %B %I %O \"%{User-Agent}i\"" crawler_bytes
CustomLog "logs/crawler_bytes.log" crawler_bytes

The columns record the serving hostname, client address, connection peer address, request time with timezone, request line, final HTTP status, response body size, received bytes, sent bytes, and reported User-Agent. The User-Agent is a request header containing the client's self-reported identification.

Apache's mod_logio documentation counts incoming bytes before decryption and outgoing bytes after encryption. For outbound transfer, select %O; for combined inbound and outbound transfer, add %I and %O. Keep %B separately for response size analysis. Existing %b logs use a hyphen when no response body bytes are recorded, whereas %B records zero.

How are crawler requests verified?

Start with the reported User-Agent to select candidate requests. For OpenAI candidates, the crawler documentation associates OAI-SearchBot with https://openai.com/searchbot.json and GPTBot with https://openai.com/gptbot.json. Compare the logged client address with the address ranges in the corresponding feed. Save the feed and its retrieval date with the report so the classification can be reviewed later.

Use separate labels for requests whose address matches the operator's published ranges and requests identified only by their User-Agent. Apply each operator's documented verification method before merging its requests into a verified crawler total.

Behind a proxy, Apache's mod_remoteip documentation explains how RemoteIPHeader replaces the logged client address. Configure RemoteIPTrustedProxy or RemoteIPInternalProxy for the actual intermediaries: Apache otherwise trusts all hosts presenting that header. %a records the resulting client address; %{c}a preserves the connection peer address.

How are the bytes aggregated?

For the chosen interval and timezone, parse each selected log record once and attach its verification label. Include rotated files covering that interval, and avoid duplicate exports or overlapping log copies. Apache's access log guide documents both rotation and configurations that write the same request to multiple logs.

Group verified requests by crawler and serving hostname. For each group, report the request count and the sum of %O; add the sum of %I when inbound transfer matters. Break down totals by final status and requested path to locate repeated requests or large responses. Keep redirects and error responses in the overall transfer total.

The choice of metric follows Apache's mod_logio definitions: %S is the combined received and sent byte count. State whether the report uses outbound bytes or combined transfer, and retain raw byte totals alongside any converted display units.

Where should a site using a content delivery network measure?

A content delivery network (CDN) introduces another measurement point. As of October 2026, Cloudflare's HTTP Requests dataset defines EdgeResponseBytes as bytes returned from its edge to the client and CacheCacheStatus as cache status. For a report about crawler-facing delivery, aggregate the edge field after verifying crawler identity using ClientIP and the reported User-Agent.

Keep origin-server and edge totals separately labelled. This distinguishes transfer measured at Apache from transfer measured by the CDN and avoids adding different legs of a request into one outbound total.

Related terms

• Server log analysis
• Crawler IP verification
• Bot bandwidth consumption
• How do I verify an OpenAI crawler request?

FAQ

Can I use an existing combined access log?

Yes, for response size analysis: Apache's combined format includes %b and User-Agent. You need a log containing %O to sum actual sent bytes using Apache's network-byte metric.

Should I include request traffic as well as responses?

Include it when your report covers both directions. Apache's mod_logio reference defines %S as %I plus %O; label that total as combined transfer.

Sources

1. Apache Software Foundation, "Log Files", accessed 17 October 2026
2. Apache Software Foundation, "Apache Module mod_log_config", accessed 17 October 2026
3. Apache Software Foundation, "Apache Module mod_logio", accessed 17 October 2026
4. OpenAI, "Overview of OpenAI Crawlers", accessed 17 October 2026
5. Apache Software Foundation, "Apache Module mod_remoteip", accessed 17 October 2026
6. Cloudflare, "HTTP Requests", accessed 17 October 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer