Which Cloudflare HTTP log fields help identify AI crawler requests?

Definition: Cloudflare HTTP request logs combine a client's reported user agent with bot classification fields and response details. As of October 2026, useful fields include ClientRequestUserAgent, VerifiedBotCategory, and, for Bot Management customers, BotScore, BotScoreSrc, and BotTags.

Key facts

• Cloudflare's HTTP Requests reference defines ClientRequestUserAgent as the user agent reported by the client.
• Cloudflare's verified bot categories distinguish legacy values including AI Crawler, AI Search, and AI Assistant.
• The bot tags documentation restricts tags to Enterprise customers who have purchased Bot Management.
• Cloudflare's datasets documentation states that field availability depends on the subscription plan.

Which fields describe the request?

The HTTP Requests field reference provides these fields for identifying requests and their targets:

Cloudflare's IP validation documentation describes verification using a bot owner's public IP list or reverse DNS, alongside a user-agent match pattern. A user-agent match alone is therefore an initial classification signal; validation also checks the request's network origin.

How should bot classification be interpreted?

A verified bot is a bot or agent whose identity and behavior Cloudflare has checked. The verified bots documentation describes AI Crawler as a legacy category for model-training crawlers, AI Assistant for user-driven bots, and AI Search for AI search services.

The same documentation describes a taxonomy introduced on July 1, 2026. It groups search crawling under Search behavior, including traditional search and AI search, and retains AI Search for compatibility with existing rules. Filtering only for the legacy AI Crawler value therefore leaves out other documented AI-related roles.

The field reference defines VerifiedBotCategory as the verified bot's category. BotScore measures automation, with scores below 30 commonly associated with automated traffic; BotScoreSrc identifies the detection engine. Both score fields require Bot Management.

Cloudflare's bot tags guide says BotTags adds detail about score assignment and can describe a verified bot or category. Its examples include google and bing.

How can the fields be selected and responses checked?

Cloudflare's API configuration guide selects exported fields through output_options.field_names. This JSON fragment shows a selection for an existing job, subject to field availability:

{
  "output_options": {
    "field_names": [
      "ClientRequestUserAgent",
      "ClientIP",
      "ClientRequestHost",
      "ClientRequestURI",
      "VerifiedBotCategory",
      "BotScore",
      "BotScoreSrc",
      "BotTags",
      "EdgeStartTimestamp",
      "RayID",
      "EdgeResponseStatus",
      "OriginResponseStatus",
      "EdgeResponseBytes",
      "SecurityAction"
    ]
  }
}

The field reference defines EdgeResponseStatus as Cloudflare's status returned to the client and OriginResponseStatus as the upstream status. EdgeResponseBytes records bytes returned to the client; SecurityAction records a terminating security-rule action, when present.

Related terms

• Server log analysis
• User-agent spoofing
• How do I compare edge and origin response codes in Cloudflare logs?
• Where do I check which AI crawlers Cloudflare recognizes?

FAQ

How can I check the field list for a zone?

Cloudflare's datasets guide documents this endpoint, with http_requests substituted for the dataset:

https://api.cloudflare.com/client/v4/zones/{zone_id}/logpush/datasets/http_requests/fields

Does an origin status of zero mean the crawler received an error?

The HTTP Requests reference says OriginResponseStatus of 0 means no origin response was received and Cloudflare's edge served the response, with a caveat for Workers subrequests. You can read EdgeResponseStatus for the status sent to the client.

Sources

1. Cloudflare, "HTTP Requests", 14 September 2026
2. Cloudflare, "Verified bots", 8 October 2026
3. Cloudflare, "Bot tags", 16 September 2026
4. Cloudflare, "Datasets", 23 April 2026
5. Cloudflare, "IP validation", 15 April 2026
6. Cloudflare, "API configuration", 21 September 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer