How do I exclude a site from future Common Crawl crawls?

Definition: As of 17 October 2026, Common Crawl documents a CCBot block in robots.txt, the file containing crawler access rules, to exclude a site from future collection by its web crawler. This control leaves previously written archive records unchanged.

Key facts

• Common Crawl's CCBot documentation supplies a crawler-specific block using User-agent: CCBot and Disallow: /.
• Its FAQ says CCBot continues checking robots.txt periodically after a block.
• Its archive format description describes WARC, or Web ARChive, records as append-only and immutable once written.

What belongs in robots.txt?

Common Crawl's CCBot page gives this configuration for preventing its crawler from crawling a website:

User-agent: CCBot
Disallow: /

The User-agent line selects CCBot. The slash in Disallow: / covers paths from the root. The crawler name in this configuration is CCBot, without the version suffix found in its request identification string.

The Robots Exclusion Protocol specification, RFC 9309, places the lowercase /robots.txt file at the service's top-level path and requires UTF-8 encoding with the text/plain media type. The specification also requires the most specific matching path rule to apply. An implementation check therefore includes reviewing existing CCBot rules for more specific Allow exceptions that would permit access under the intended block.

How can the block be checked?

The Common Crawl FAQ states that CCBot checks robots.txt before fetching permitted pages. A practical check starts with the publicly served /robots.txt: its response must contain the intended CCBot group. Continued requests for that file are consistent with Common Crawl's documented periodic checks after exclusion.

Common Crawl's crawler documentation warns that other crawlers falsely identify themselves as CCBot. It provides dedicated IP address ranges and a reverse-DNS verification example. For an IPv4 request, the example resolves the address to a name under crawl.commoncrawl.org, then resolves that name back to the original address. The same documentation supplies a JSON list of address ranges and notes that reverse DNS is unavailable for its IPv6 addresses.

What happens to data already archived?

Common Crawl's About page describes WARC records as raw crawl data containing full HTTP responses and says they are immutable once written. It also says the dataset is publicly downloadable.

The practical consequence is that changing the site's robots.txt leaves earlier archive records in place. A website's new crawler rule also supplies no deletion instruction to third parties holding downloaded copies. Checking the current crawler rule and addressing existing copies are separate tasks.

Where do separate removal requests go?

Common Crawl's Opt-Out Registry announcement, published on 17 September 2025, describes a public list of legal opt-out requests, now called the Opt-Out Ledger. Its stated purpose includes alerting downstream users to material that content owners request be excluded from crawls. It directs questions and requests to info@commoncrawl.org.

For personal data, Common Crawl's Privacy Policy asks requesters to provide full website URLs so it can locate the information. It describes reasonable efforts to process requests and possible retention for legal reasons or legitimate business purposes. Submitting a request is a separate procedure from deploying the CCBot rule.

Related terms

• CCBot (Common Crawl)
• robots.txt
• robots.txt group precedence

FAQ

Do I need to email Common Crawl before blocking CCBot?

Common Crawl's CCBot documentation directly provides the robots.txt block. Its opt-out announcement separately offers an email route for requests.

Can I use Crawl-delay to exclude my site?

The Common Crawl FAQ documents Crawl-delay as a way to slow requests. To exclude your site, use its documented Disallow: / rule for CCBot.

Sources

1. Common Crawl, "CCBot", undated
2. Common Crawl, "Frequently Asked Questions", undated
3. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
4. Common Crawl, "About", undated
5. Common Crawl, "Common Crawl Foundation Opt-Out Registry", 17 September 2025
6. Common Crawl, "Privacy Policy", updated 3 April 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer