In short. CCBot is Common Crawl’s web crawler, with documented robots.txt controls and published address ranges for verifying requests.
Definition: CCBot is the web crawler operated by Common Crawl, a nonprofit organization that makes web crawl archives freely available for research and analysis. Its data contributes to datasets used to train large language models, which are models trained on text to process and produce language.
Key facts
- As of 17 October 2026, Common Crawl's CCBot documentation identifies the crawler with the user-agent string
CCBot/2.0 (https://commoncrawl.org/faq/). - Common Crawl's FAQ states that CCBot checks
robots.txtbefore fetching allowed pages through HTTP GET requests. - The same FAQ documents that CCBot neither executes JavaScript nor uses cookies.
- Common Crawl's CCBot page warns that other crawlers falsely identify themselves as CCBot and supplies methods for verifying requests.
What happens to pages collected by CCBot?
Common Crawl's About page describes an open repository whose data is used, after filtering and processing by third parties, to train large language models. For a website, allowing CCBot permits collection for that public repository and its downstream uses.
The corpus overview distinguishes raw web page data from metadata extracts and text extracts. Access to the corpus hosted by Amazon is free, and users can download all or part of it. The same overview links to the Common Crawl URL Index for searching for archived pages.
The FAQ describes the dataset as a sample of the web, generally containing a randomly selected subset of a website. A crawler visit therefore concerns collection of particular pages within that sample.
How does a website control CCBot?
A robots.txt file provides crawling instructions. Common Crawl's blocking instructions give this exact group for preventing CCBot from crawling a website:
User-agent: CCBot
Disallow: /
The FAQ states that CCBot periodically checks the file for updates even after crawling is blocked. It also documents Crawl-delay, a setting that requests a pause between fetches. Its example limits requests to at most one every two seconds:
User-agent: CCBot
Crawl-delay: 2
These are alternative configurations for blocking crawling or slowing it. The FAQ also says CCBot uses sitemaps announced in robots.txt. A sitemap is a file listing website URLs for crawlers.
For requests beyond crawler configuration, Common Crawl's Opt-Out Registry announcement describes its Opt-Out Ledger, the current name for its published list of legal opt-out requests. The stated purpose includes informing downstream users about material that content owners request to exclude from crawls.
How can a website verify a CCBot request?
The user-agent is the identity string sent with a request. Common Crawl's verification guidance shows an IPv4 address resolving through reverse DNS to a hostname under crawl.commoncrawl.org, followed by a forward lookup that returns the original address. DNS is the system that maps domain names to network addresses; this check compares both directions of that mapping.
Common Crawl also publishes its address ranges in a JSON feed. Its CCBot documentation states that reverse DNS is unavailable for its IPv6 addresses, so the published ranges provide the applicable verification route for those requests.
Related terms
FAQ
Can I slow CCBot without blocking it?
Yes. Common Crawl's FAQ documents the Crawl-delay setting and gives a two-second interval as an example.
How can I look for pages already collected?
You can search the Common Crawl URL Index linked from the corpus overview. The overview describes the index as a way to search for pages in the corpus.
Where can I send an opt-out request?
Common Crawl's opt-out announcement gives info@commoncrawl.org for questions about opt-out procedures and for submitting requests.
Sources
- Common Crawl, "CCBot", undated
- Common Crawl, "Frequently Asked Questions", undated
- Common Crawl, "About", undated
- Common Crawl, "Overview", undated
- Common Crawl, "Common Crawl Foundation Opt-Out Registry", 17 September 2025
- Common Crawl, "CCBot IP ranges", 11 August 2026
Check how a page reads to answer engines. Open the GEO score analyzer