In short. Cloudflare verifies Web Bot Auth requests using registered Ed25519 public keys, signed message components and required signature headers.
Definition: Web Bot Auth authenticates automated HTTP requests through cryptographic signatures. In Cloudflare's implementation, a crawler registers public keys and signs requests with the corresponding private key so Cloudflare can verify the sender's identity.
Key facts
- As of 17 October 2026, Cloudflare's Web Bot Auth documentation specifies Ed25519 signing keys and registration through the Request Signature verification method.
- The IETF's HTTP Message Signatures standard, RFC 9421, defines how a verifier reconstructs the signed data from the received message and signature parameters.
- Cloudflare's verified bots policy requires honest self-identification and non-abusive behavior, including respect for robots.txt and crawl directives.
What does the crawler operator register?
A public key lets a verifier check a signature produced with its corresponding private key. Cloudflare's integration instructions require the operator to publish a JSON Web Key Set, a JSON collection of public keys, over HTTPS at this path:
/.well-known/http-message-signatures-directory
The directory response uses the content type application/http-message-signatures-directory+json and carries a signature for each published key. Its signature parameters include the tag http-message-signatures-directory and a keyid containing the key's JWK thumbprint, an identifier calculated from the public key. Cloudflare explains that signing the directory response prevents another party from mirroring it and registering on the operator's behalf.
Registration takes place under Manage Account > Configurations > Bot Submission Form. The operator selects Request Signature and supplies the key directory URL under Validation Instructions. Signing crawler requests follows successful verification of the bot.
How is a request signed and checked?
A signed component is a selected HTTP header or a value derived from the request. Cloudflare's request-signing instructions recommend covering @authority, which represents the destination domain. They also require covering the Signature-Agent header, whose value identifies the key directory using a quoted HTTPS URI. Cloudflare's example uses:
Signature-Agent: "https://signature-agent.test"
The request also carries Signature-Input, which lists the covered components and signature parameters, and Signature, which contains the signature value. The parameters include tag="web-bot-auth", keyid, created and expires. The key identifier is the thumbprint of the registered public key; the timestamps describe creation and expiry.
Under RFC 9421's verification procedure, the verifier reconstructs the signature base, the sequence of bytes representing the covered message components and signature parameters. It then applies the cryptographic verification algorithm using that reconstructed base, the verification key and the received signature. Cloudflare implements this mechanism with its registered Ed25519 keys and its documented Web Bot Auth requirements.
A replay attack reuses a previously signed message. Cloudflare recommends short expiry intervals and says a minute is often sufficient. It recommends a nonce, a per-message value intended to help prevent replay, but currently performs no nonce validation or tracking of previously seen nonces.
How can verification failures be checked?
Cloudflare's test endpoint documentation identifies this destination for signed test requests:
https://crawltest.com/cdn-cgi/web-bot-auth
It returns HTTP 200 when the key is known and the message verifies. HTTP 401 can indicate an unknown key despite correct formatting, or a known key with failed verification; other failures return HTTP 400.
The same documentation identifies concrete failure conditions: an unquoted Signature-Agent URI, the dictionary syntax used in later drafts, an unsigned Signature-Agent header, expired timestamps and unsupported components. Cloudflare requires ASCII component values and does not support the bs or sf component parameters.
Related terms
- Reverse DNS bot verification
- User-agent spoofing
- How do I verify an OpenAI crawler request?
- How do I check whether a CDN blocked an AI crawler?
FAQ
Where can you use Cloudflare's verified-bot result?
Cloudflare's field reference defines cf.bot_management.verified_bot as a Boolean indicating a known good bot or crawler. The field requires an Enterprise plan with Bot Management enabled and provides the same information as cf.client.bot.
Can you turn off Cloudflare's signature verification for origin processing?
Cloudflare's troubleshooting documentation directs you to contact Cloudflare Support to disable Web Bot Auth verification for a zone. Cloudflare then stops validating incoming signatures, and verified bots fall back to other validation methods, such as reverse DNS.
Sources
- Cloudflare, "Web Bot Auth", updated 8 October 2026
- IETF, "RFC 9421: HTTP Message Signatures", February 2024
- Cloudflare, "Verified bots", updated 8 October 2026
- Cloudflare, "cf.bot_management.verified_bot", undated
Check how a page reads to answer engines. Open the GEO score analyzer