In short. Bot response code distribution groups crawler requests by response outcome to measure successful fetches, redirects and failures.

Definition: Bot response code distribution is the breakdown of crawler requests by the response or failure recorded for each request. It expresses outcomes as counts or percentages, separating successful responses, redirects and errors.

Key facts

As of 17 October 2026, Google's Crawl Stats documentation describes response percentages calculated from requests; repeated requests to one URL count separately.
Apache's Log Files documentation identifies %>s as the response status recorded in its access log.
Google's HTTP status documentation states that 5xx responses and 429 temporarily slow its crawlers.
Google's request verification guidance provides address checks for establishing whether crawler traffic comes from Google.

What does the distribution count?

Google's Crawl Stats report groups responses as a percentage of crawl responses. Its example requests the same URL twice: a 500 followed by a 200 produces an equal split between server error and OK. Each request in a server-side redirect chain also counts separately.

For a summary calculated from logs, the corresponding arithmetic is:

response share = requests with that response / requests in the selected set

The selected set needs an explicit crawler, reporting period and host scope. This is a calculation derived from Google's request-based definition. Counts alongside shares preserve the request volume behind each percentage.

How should successful and failed responses be interpreted?

Google's HTTP status guidance describes 2xx as success and 3xx as redirection. A 200 response passes content to further processing. An empty page or an error message can still produce a soft 404, Google's classification for error-like content received with a success status.

A 429 signals an overloaded server to Google's crawlers. It belongs alongside 5xx in an investigation of reduced crawling, even though its status code begins with a different digit. Keeping individual codes visible preserves this distinction when a report also groups responses by class.

Google's Crawl Stats documentation lists 304, meaning unchanged since the previous crawl, among good responses. It also explains that 404 can be appropriate for a permanently removed page without a replacement. Interpretation therefore depends on the requested resource and the intended response.

How can server logs support the summary?

Apache's access log documentation describes fields for the client address, request time and response status. Its Combined Log Format adds the User-Agent request header, the identifying information reported by the client. An adapted format using these documented fields is:

LogFormat "%h %t %>s \"%{User-agent}i\"" bot_status
CustomLog "logs/access_log" bot_status

The status field supports grouping outcomes; the address and User-Agent fields support selecting crawler requests. For Google traffic, Google's verification instructions describe matching addresses against published IP ranges or using reverse and forward domain name lookups. The forward lookup must return the original address from the log.

FAQ

Why can Crawl Stats show failures absent from my access logs?

Google's Crawl Stats documentation includes a category for requests that never reach the server. It also counts potential fetches abandoned because robots.txt is unavailable, which can make crawl totals exceed logged requests.

Can I use the status code alone to identify crawler traffic?

Apache's log documentation assigns the response outcome to %>s and the client's reported identity to the User-Agent field. Google's verification guidance adds address checks for traffic claiming to come from Google.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer