How can an owned monitoring bot be excluded from AWS WAF Bot Control?

Definition: AWS WAF documents a shared-secret request header as a way to exclude a site monitoring bot or custom bot from Bot Control inspection. The exception is expressed with a scope-down statement on the Bot Control managed rule group.

Key facts

• AWS's custom bot example configures a monitoring or custom bot to send a shared secret in a header, then excludes requests carrying that header from Bot Control inspection.
• A scope-down statement narrows the requests evaluated by a containing managed rule group. Requests that do not match the scope-down statement are treated as not matching the rule group.
• The example uses the AWS managed rule group named AWSManagedRulesBotControlRuleSet and sets its inspection level to COMMON, as shown in the Bot Control example.
• AWS's managed rule group reference lists CategoryMonitoring for bots used to monitor performance or uptime and describes a rule action that applies to unverified bots.

How does the documented exception work?

The bot sends a shared secret in a custom request header. AWS's example names that header x-bypass-secret and compares it with a byte-match statement. The statement uses EXACTLY as its positional constraint and applies no text transformation. The outer NotStatement reverses the byte-match result, so a request with the example secret falls outside the requests sent to Bot Control, while requests that fail the exact comparison remain in scope.

AWS defines a scope-down statement as a nestable rule statement added inside a managed rule group statement or a rate-based statement. The containing rule evaluates requests that match the scope-down statement, and AWS WAF treats requests that fail that match as not matching the managed rule group.

What does the AWS example contain?

The source example is a JSON rule named AWS-AWSBotControl-Example with priority 5. Its managed rule group identifies vendor AWS, uses AWSManagedRulesBotControlRuleSet, and sets InspectionLevel to COMMON. The header match uses the encoded search string YSBzZWNyZXQ=, the single header name x-bypass-secret, text transformation priority 0 with type NONE, and positional constraint EXACTLY.

{
  "Name": "AWS-AWSBotControl-Example",
  "Priority": 5,
  "Statement": {
    "ManagedRuleGroupStatement": {
      "VendorName": "AWS",
      "Name": "AWSManagedRulesBotControlRuleSet",
      "ManagedRuleGroupConfigs": [
        {
          "AWSManagedRulesBotControlRuleSet": {
            "InspectionLevel": "COMMON"
          }
        }
      ],
      "RuleActionOverrides": [],
      "ExcludedRules": []
    },
    "VisibilityConfig": {
      "SampledRequestsEnabled": true,
      "CloudWatchMetricsEnabled": true,
      "MetricName": "AWS-AWSBotControl-Example"
    },
    "ScopeDownStatement": {
      "NotStatement": {
        "Statement": {
          "ByteMatchStatement": {
            "SearchString": "YSBzZWNyZXQ=",
            "FieldToMatch": {
              "SingleHeader": {
                "Name": "x-bypass-secret"
              }
            },
            "TextTransformations": [
              {
                "Priority": 0,
                "Type": "NONE"
              }
            ],
            "PositionalConstraint": "EXACTLY"
          }
        }
      }
    }
  }
}

What should you check before using it?

Use the header name and secret consistently in the bot request and the scope-down match. The example's exact comparison means a different value does not satisfy the inner byte-match statement. AWS says that a scope-down statement can use any nestable rule, and that its text transformations and forwarded IP settings apply to the scope-down statement itself rather than being inherited by the containing managed rule group.

The example selects the COMMON inspection level. AWS documents COMMON as detecting self-identifying bots and high-confidence bot signatures, while TARGETED includes the common protections and adds detection for sophisticated bots that do not self-identify. The managed rule reference also says that verified bots are not blocked by the Bot Control managed rule group, while the CategoryMonitoring rule applies its block action to unverified bots.

Related terms

• How do AWS WAF Bot Control inspection levels differ?
• How do AWS WAF Bot Control rules treat verified bots?
• When can a later AWS WAF rule match a label added by another rule?

FAQ

Does the header exception use robots.txt?

No. The AWS example is a JSON AWS WAF rule with a scope-down statement attached to the Bot Control configuration. It operates on a request header.

What happens to a request without the shared secret?

The inner byte-match statement does not match the required value, so the outer NotStatement keeps that request within the scope evaluated by Bot Control. AWS describes the same scope-down behavior as evaluating only requests that match the scope-down statement.

Can the same pattern use targeted inspection?

AWS's example sets InspectionLevel to COMMON. The managed rule reference documents TARGETED as the other protection level, with additional detection for sophisticated bots that do not self-identify.

Sources

1. AWS, "Bot Control example: Allowing traffic from a bot that you control", n.d.
2. AWS, "Using scope-down statements in AWS WAF", n.d.
3. AWS, "AWS WAF Bot Control rule group", n.d.

---

Check how a page reads to answer engines. Open the GEO score analyzer