Does Googlebot support Crawl-delay?

Definition: Googlebot does not support Crawl-delay in robots.txt, the file that tells crawlers which URLs they may access. As of 17 October 2026, Google's specification lists user-agent, allow, disallow, and sitemap as supported fields and explicitly excludes crawl-delay.

Key facts

• Google's robots.txt specification lists four supported fields; adding Crawl-delay supplies no supported delay setting for Googlebot.
• Google's introduction to robots.txt explains that a disallowed page's URL can still appear in Search if other pages link to it.
• Google's crawl-rate guidance recommends checking recent server access logs and consulting the hosting company when crawling increases sharply.

Which robots.txt fields does Google support?

The syntax section of Google's specification gives each supported field a distinct role:

Field names are case-insensitive. The values of allow and disallow are case-sensitive paths, and rules with no path are ignored. A sitemap value includes the protocol and hostname. The sitemap field applies independently of the crawler groups in the file.

How does a supported configuration work?

Google's specification includes this configuration, reproduced here without its comments:

User-agent: *
Disallow: /includes/

User-agent: Googlebot
Allow: /includes/

Sitemap: https://example.com/sitemap.xml

The asterisk identifies the general crawler group. Googlebot follows its specific group, which permits crawling /includes/. Google selects the most specific matching user-agent group; a specific group and the general * group are not combined. The sitemap line supplies a sitemap location outside that group selection.

For conflicting path rules within the applicable group, Google uses the longest matching path. If equally specific rules conflict, it uses the least restrictive rule. These settings determine access to paths; none supplies a request interval.

How can a site reduce Google's crawl traffic?

Google's crawl-rate guidance identifies faceted navigation, meaning URL variations for filters or sorting, as a possible cause of excess crawling. It recommends examining access logs to identify the traffic source.

For an emergency lasting a few hours or one to two days, the guidance documents returning HTTP 500, 503, or 429 responses to crawl requests. Significant numbers of these responses reduce crawling across the hostname, including URLs that still return content. Crawling increases again as the errors decline.

This emergency measure has consequences: existing pages refresh less frequently in Search, and URLs returning these errors for multiple days may leave the index. It requires a temporary server response change; adding Crawl-delay provides no such control.

Related terms

• robots.txt
• Googlebot
• Crawl-delay
• robots.txt group precedence

FAQ

Does writing crawl-delay in lowercase help?

No. Google's specification treats field names as case-insensitive and explicitly lists crawl-delay as unsupported. Changing capitalization adds no delay control.

Can I use Disallow to keep a page out of Google Search?

Google's robots.txt introduction explains that a disallowed URL can still appear when linked elsewhere. If you need to keep a page out of Search, the introduction points to password protection or an indexing control such as noindex.

Sources

1. Google, "How Google interprets the robots.txt specification", 31 August 2026
2. Google Search Central, "Introduction to robots.txt", 10 December 2025
3. Google, "Reduce the Google crawl rate", 6 October 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer