In short. Bot bandwidth consumption measures transferred bytes for automated requests, with costs depending on the observation point and billing model.

Definition: Bot bandwidth consumption is the volume of data transferred in automated HTTP requests and their responses during a stated period. A measurement specifies the bot group, the observation point and whether the byte count includes headers or encryption overhead.

Key facts

The field definitions and billing examples below reflect Apache and AWS documentation as of 17 October 2026.

Apache's log format documentation defines %B and %b as response sizes excluding HTTP headers.
Apache's mod_logio documentation defines %I as bytes received and %O as bytes sent, including headers, with encryption reflected in the counts.
AWS's CloudFront logging reference defines sc-bytes as bytes sent to the viewer, including headers.
AWS's pay-as-you-go pricing charges separately for outgoing data transfer and HTTP or HTTPS requests.

Which Apache fields measure transferred bytes?

Apache's mod_log_config documentation distinguishes response size from actual transfer: aborted connections and SSL can make those values differ. %b records a hyphen when no bytes are sent; %B records a numeric value. Reports using %b need to handle that hyphen before summing sizes.

For network transfer accounting, Apache's mod_logio module supplies %I, %O and %S. %S combines incoming and outgoing bytes. Counting happens before SSL/TLS processing on input and after SSL/TLS processing on output. On an SSL KeepAlive connection, the handshake overhead belongs to the first request's byte count.

How can a website build a bot usage report?

A custom access log can retain identity fields alongside response size, transferred bytes and elapsed time. This configuration combines fields defined in Apache's log format documentation; %I and %O require mod_logio to be enabled:

LogFormat "%a %t \"%r\" %>s %B %I %O %D \"%{User-agent}i\"" bot_usage
CustomLog "logs/bot_usage.log" bot_usage

Here %a records the client address, %t the request time, %r the request line and %>s the final status. %D measures elapsed service time in microseconds. The report can group matching records by bot and reporting period, count requests, and sum incoming and outgoing bytes separately. Keeping the response status makes it possible to break out error responses.

Bot attribution needs verification. Google's crawler verification documentation, updated 20 March 2026, describes matching addresses against published IP ranges or checking reverse DNS followed by forward DNS. A report should label a group selected only by its User-Agent header as a claimed identity.

Where should a CDN-backed website measure traffic?

A content delivery network (CDN) serves requests through edge servers; the origin is the website server behind it. AWS's CloudFront logging reference distinguishes Hit, where the edge serves cached content, from Miss, where it forwards the request to the origin.

Consequently, an origin access log omits requests served entirely from the edge cache. An edge report can sum sc-bytes for outgoing data and cs-bytes for incoming data. Origin transfer and edge delivery remain separate totals because they describe different network legs. Adding both together requires a report explicitly measuring traffic across both legs.

How does usage become a cost estimate?

AWS's CloudFront pay-as-you-go pricing lists outgoing transfer per GB and request prices per 10,000 requests. Rates depend on usage type, region and enabled features. A bot cost estimate therefore applies the relevant transfer rate to attributable billable volume and the relevant request rate to attributable billable requests, accounting for allowances and pricing tiers.

The billing plan matters: AWS's flat-rate pricing page describes monthly plans with no overage charges. A byte total remains a usage measurement under that model; assigning part of the monthly fee to bots is an allocation choice.

FAQ

Can I use an existing combined access log?

Apache's combined log example contains %b, so you can total response sizes after handling hyphens. For actual outgoing transfer, add %O with mod_logio enabled.

Does a cached response consume bandwidth?

Yes. AWS's logging reference defines Hit as delivery from cache to the viewer, and sc-bytes measures the bytes sent in that response.

Is elapsed request time a compute bill?

The same Apache field documentation defines %D as time taken to serve the request in microseconds. Converting that duration into a compute charge requires the hosting provider's billing model.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer