CCBot
Operated by Common Crawl · Builds an open archive
If you block it
Your pages stop entering the open archive.
What Common Crawl says it is for
Common Crawl describes itself as a non-profit producing and maintaining an open repository of web crawl data, so that web data is freely available for research.
Published details
- robots.txt
- Treats robots.txt as the opt-out, without saying it respects it“The page gives you an opt-out rule to add to robots.txt, but never states in general terms that CCBot respects the file.”
- User agent
CCBot/2.0 (https://commoncrawl.org/faq/)Match on the prefixCCBot/rather than the whole string: operators version these, and an exact match stops working on the next release.- Verification
- Reverse DNS to
*.crawl.commoncrawl.org, then forward-resolve that name back to the same address. - Published address list: https://index.commoncrawl.org/ccbot.json
- Reverse DNS to
Not addressed in the documentation
- As of , Common Crawl’s documentation does not address AI training — the page does not mention it at all, in either direction.
Third-party reports
Not published by Common Crawl. Each statement is attributed to its source.
- The archive is widely described as the single largest source of training data for large language models, and most model providers have trained on it. — Widely reported; this is not a statement by Common Crawl.
Source: https://commoncrawl.org/ccbot Read .