Organization
Common Crawl
Common Crawl is a data services organization based in Beverly Hills, California26. This catalog lists 1 dataset from Common Crawl, under licenses including the Common Crawl Terms of Use (limited, non-exclusive license).
Documented roles: Data services
Common Crawl is a nonprofit that crawls the web and makes its archives and derived datasets freely available. It has collected crawl data since 2008 and hosts it on Amazon Web Services through an AWS open data sponsorship program.13
Using this organization’s offerings
The detailed audience, self-hosting, pricing, and restrictions review has not yet been completed. Consult the sourced product records below and the publisher’s current terms. Missing detail means unknown, not unavailable.
Key facts
What is and isn’t public
Products
No product records yet.
Cataloged artifacts
Datasets
- Common Crawl corpusDatasetAvailability: Public
Notable documented facts
- Common Crawl's crawler, CCBot, checks robots.txt before fetching pages and honors the Crawl-delay parameter and nofollow attributes; site owners can block it through robots.txt, and Common Crawl also refers site owners to an opt-out registry.45
- The Terms of Use (last updated March 7, 2024) require users to indemnify Common Crawl against third-party claims arising from their use of the service or crawled content, expressly including use in connection with artificial intelligence and machine learning systems such as large language models.2
In the news
Common Crawl releases experimental web graph embeddings for web hosts
On September 7, 2026, Common Crawl published an experimental dataset of 128-dimensional embeddings for web hosts, learned from hyperlinks alone in its host-level web graph for the November 2025, December 2025, and January 2026 crawls. The model uses no page text, metadata, or hostnames and was trained only on hosts with at least eight links. The data is published as Parquet files on Common Crawl's S3 bucket and on Hugging Face, where the dataset card says its use is subject to the Common Crawl Terms of Use; the source code is on GitHub. Common Crawl says it is yet to be seen whether it will publish graph embeddings on a regular basis.
What this catalog does not know
- Official careers page: not verified by this catalog.
- Products: none documented in this record.
Have a primary source? How to report a correction.
U.S. eligibility
Eligible · basis: U.S. nonprofit or lab
Common Crawl describes itself as a 501(c)(3) nonprofit. Its Terms of Use name The Common Crawl Foundation with a Beverly Hills, California address, and the IRS exempt-organization extract for California lists Commoncrawl Foundation at the same address as a 501(c)(3) organization.126
Sources
This listing is not an endorsement or a federal approval.
Share or cite this page
- X (opens X)
- Bluesky (opens Bluesky)
- LinkedIn (opens LinkedIn)
- Reddit (opens Reddit)
- Hacker News (opens Hacker News)
- Email (opens your email app)
- RSS updates
Cite this page
APA style
USASI. (2026, October 1). Common Crawl. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/companies/common-crawl/
BibTeX
@misc{usasi_common_crawl,
author = {{USASI}},
title = {Common Crawl},
year = {2026},
month = oct,
howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/companies/common-crawl/}},
note = {United States of America Superintelligence. Last updated 2026-10-01}
}The date is when this page was last updated. Add the date you read it if your style needs one.
For maintainers: a “Listed in USASI” badge
Links to this page. It says the entry is listed, not endorsed or rated. About the badge
Markdown (for a README)
[](https://unitedstatesofamericasuperintelligence.com/companies/common-crawl/)
HTML
<a href="https://unitedstatesofamericasuperintelligence.com/companies/common-crawl/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>