Are AWS WAF user-triggered verified bots treated like verified search crawlers?

The Citead team, updated

In short. AWS WAF treats user-triggered and developer-platform verified labels like unverified bots, unlike ordinary verified search-engine bots.

Definition: No. AWS WAF gives user-triggered and developer-platform bots labels that contain verified, but its Bot Control documentation says both categories are treated like unverified bots. A verified search-engine bot follows the ordinary verified-bot treatment in the category rules.

Key facts

  • AWS defines separate bot:user_triggered:verified and bot:developer_platform:verified labels, and says both categories are treated like unverified bots in its Bot Control managed rule reference.
  • AWS's common protection level labels self-identifying bots and verifies generally desirable bots, according to its Bot Control overview.
  • The AWS WAF labeling documentation says a label remains available to rules that run later in the web ACL evaluation.

Which labels are being compared?

AWS's Bot Control managed rule reference uses bot:verified for a bot that identifies itself and that Bot Control can verify. The reference says this label can be combined with a category such as bot:category:search_engine or a name such as bot:name:googlebot.

The two similar categories have their own full labels. The word verified in those labels describes the category name, while the documented rule treatment remains the one AWS assigns to that category:

awswaf:managed:aws:bot-control:bot:verified
awswaf:managed:aws:bot-control:bot:category:search_engine
awswaf:managed:aws:bot-control:bot:user_triggered:verified
awswaf:managed:aws:bot-control:bot:developer_platform:verified

user_triggered:verified identifies a bot that is similar to a verified bot but might be directly invoked by end users. developer_platform:verified identifies a similar bot used by developer platforms for scripting. AWS gives Google Apps Script as the example for the second category.

How does the search-engine comparison work?

The managed rule reference describes CategorySearchEngine as inspecting bots that crawl websites to index content and make information available for search engine results. Its listed action is Block for unverified bots. For verified bots, the rule group does not match that rule and takes no action, while adding the bot name, category, and bot:verified labels.

That is the ordinary verified-bot path. AWS separately says that user-triggered and developer-platform categories are treated by Bot Control rules like unverified bots. Their label names therefore do not give them the search-engine category's verified-bot exemption.

What can a web ACL do with these labels?

Bot Control adds labels to evaluated requests, and AWS's Bot Control overview says those labels can be matched in custom AWS WAF rules. The general labeling documentation states that a matching label stays available until web ACL evaluation ends, so a later label match rule can use the Bot Control result.

The managed group itself does not block the set of commonly allowed verified bots. AWS says a custom rule can use the labels to block those bots or a subset of them. A web ACL can therefore create separate handling for the full user-triggered label, the developer-platform label, or a more specific combination with a bot name or category.

FAQ

Does verified in the label mean the bot gets the verified search-crawler treatment?

No. AWS uses distinct labels for the user-triggered and developer-platform categories and says both are treated like unverified bots.

Is a verified search-engine bot treated the same way?

No. For the CategorySearchEngine rule, AWS says verified bots do not match the rule and receive no action from it, while the rule group adds their bot and verification labels.

Can I handle the two categories with a later custom rule?

Yes. Bot Control labels remain available to later rules during web ACL evaluation, and AWS documents custom label matching for handling requests.

Sources

  1. AWS, "AWS WAF Bot Control rule group", undated
  2. AWS, "Web request labeling in AWS WAF", undated
  3. AWS, "AWS WAF Bot Control", undated

Check how a page reads to answer engines. Open the GEO score analyzer

Test a page and see what to fix first

The analysis usually takes less than 10 seconds and needs no account.

400,000+ pages scored in the public benchmark