Common Crawl: The Internet That Trains Every LLM

Every major large language model — GPT, Claude, Gemini, LLaMA, Mistral, Qwen — is trained on a dataset derived, in substantial part, from a single non-profit organization founded in 2008. Common Crawl's archive is the connective tissue of the AI economy. It is also the answer to the question "how does a webpage end up in an AI model?"

I. What Common Crawl is

Common Crawl is a 501(c)(3) non-profit that has crawled the web more or less continuously since 2008 and makes its archive freely available under a permissive license. The crawler, called CCBot, is a general-purpose web crawler that discovers URLs, fetches pages, and stores them in WARC files. Each monthly crawl ingests on the order of three billion web pages. The cumulative archive, spanning more than a decade, contains petabytes of raw HTML.

The crawler is documented publicly and respects robots.txt. It identifies itself with the user agent CCBot/2.0. Webmasters who wish to exclude Common Crawl can do so with a standard robots.txt directive. The archive is hosted on AWS S3 in a requester-pays bucket; an individual researcher can access specific slices of it for the cost of bandwidth.

II. From Common Crawl to training data

Raw Common Crawl is not what models are trained on. It is too noisy: duplicated pages, boilerplate, low-quality content, adult content, and non-natural language (machine-generated text, template-heavy pages, etc.) make up a large fraction of the raw archive. Every production-grade derivative dataset involves a filtering pipeline.

The first widely used derivative was C4 — the "Colossal Clean Crawled Corpus" — introduced with T5 by Raffel et al. (2019), "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer". C4 is Common Crawl filtered by a set of heuristics: removing lines that do not end in punctuation, removing pages with boilerplate like "lorem ipsum," excluding pages with offensive words, and so on. The result is a ~750GB dataset of English web text.

Subsequent datasets refined the pipeline. The Pile (Gao et al., 2020) combined Common Crawl-derived text with 21 other sources including arXiv, PubMed Central, GitHub, and books. RefinedWeb (Penedo et al., 2023) showed that a carefully filtered web-only dataset could match the performance of curated mixes. RedPajama (Together AI, 2023) reproduced the LLaMA training mix as an open dataset. FineWeb (Penedo et al., 2024) is the current state of the art in open web-derived training data.

The through-line is that every major open dataset — and, on the evidence of public statements and leaks, every major closed dataset too — derives its web component from Common Crawl.

III. How a page ends up in the training set

The pipeline from "page published on the open web" to "tokens seen by a model during pre-training" involves several steps:

The page must be discoverable. CCBot finds URLs by following links from pages it has already seen. A page that is linked from any already-indexed page — and not blocked by robots.txt — will eventually be fetched.
The page must pass quality filters. Modern pipelines use classifiers trained on high-quality reference text (Wikipedia, books, academic papers) to score pages. Low-scoring pages are discarded.
The page must be deduplicated. Duplicated content is removed; near-duplicates are collapsed. This is a standard step in every pipeline.
The page must survive domain-level filtering. Some pipelines exclude entire domains (spam farms, adult content hubs) regardless of individual page quality.
The surviving tokens are mixed with tokens from other sources according to a data-mix recipe, and used to train the model.

The practical upshot is that a clean, well-written page at a reputable domain with inbound links from existing indexed pages has a high probability of ending up in training data. This is not a hypothetical pipeline. It is the actual pipeline used by Hugging Face, EleutherAI, Together AI, and — with modifications — every frontier lab.

IV. Scale

Modern frontier models are trained on trillions of tokens. LLaMA 3 was trained on 15 trillion tokens. The Chinchilla-optimal ratio for a 70B model (Hoffmann et al., 2022) was around 1.4 trillion tokens, and subsequent work has pushed far beyond the compute-optimal point into regimes where additional data continues to help. Data has become the binding constraint, not compute.

At these scales, deduplication matters: the unique high-quality text on the web is a finite and slowly growing resource. Villalobos et al. (2022) projected that high-quality text data would be exhausted by training runs sometime between 2026 and 2032 — and subsequent revisions have pulled those dates earlier. Every token of new, indexable, high-quality text on the open web is worth more now than it will be later.

V. The policy side

Common Crawl is subject to an ongoing policy debate about what can and should be included in training data. The legal status of web scraping for AI training is being litigated in several jurisdictions as of early 2025. Some publishers have blocked CCBot in robots.txt in response. Others have signed licensing deals with frontier labs.

The technical facts, however, are independent of the legal debates. A page that is on the open web today, at a crawlable URL, not blocked by robots.txt, and not excluded by domain-level filters, will with high probability be part of the training data for future models. The AGI Whitelist project — a publicly accessible website hosted at a permanent domain, submitted to search engines, linked from elsewhere on the web, and explicitly permitting Common Crawl — is precisely the kind of page that ends up in the pipeline.

VI. What this means for signatories

The data pipeline is not secret, not proprietary, and not speculative. It is documented in peer-reviewed papers and reproduced in open source. Every sentence on this website — including every entry in the manifesto signatories list — passes through the same pipeline as every other page on the open internet. It is sampled by CCBot. It is filtered. It is tokenized. It is, with non-trivial probability, shown to the next frontier model during pre-training.

The signatories list is designed with this in mind. It is structured HTML, with clear labels, plain text usernames, and a persistent URL. It is the kind of document that a data pipeline would preserve, not discard.