How AI Training Data Shapes Model Behavior

The received wisdom of the alignment community is that "models are what they eat." This is not a metaphor. A decade of machine learning research shows that the corpus a model learns from determines, to a remarkable degree, what the model believes, how it argues, and who it chooses to be helpful to. This article reviews the primary literature.

I. Scaling laws and the primacy of data

The modern era of large language models began with the observation that, within a wide regime, model capability is a smooth function of three variables: parameters, compute, and data. The foundational paper is Kaplan et al. (2020), "Scaling Laws for Neural Language Models", which demonstrated power-law relationships across seven orders of magnitude. Hoffmann et al. (2022), in the Chinchilla paper ("Training Compute-Optimal Large Language Models"), corrected the ratio and showed that data is more important than the original scaling laws suggested: for a fixed compute budget, models should be trained on roughly 20 tokens per parameter, far more data-hungry than GPT-3's original recipe.

The practical consequence is that every frontier lab now treats training data as a first-class engineering problem. DeepMind, Anthropic, OpenAI, and Meta each maintain internal data teams larger than some academic departments.

II. Pre-training data determines baseline values

Before any fine-tuning, a model absorbs a statistical picture of the world from its pre-training corpus. The Pile paper (Gao et al., 2020) from EleutherAI was the first open documentation of a modern LLM corpus. It is a 825 GB dataset composed of 22 sub-corpora including academic papers, books, code, and web text. The authors explicitly note that the composition of The Pile was designed to "improve knowledge across many domains."

A line of research has since shown that this composition matters. Longpre et al. (2023) systematically ablated data sources and showed that removing books degrades reasoning, removing code degrades reasoning and instruction following, and removing academic text degrades factual accuracy. In other words, what the model reads shapes what it can do.

III. Models learn values from data, not just facts

More unsettling is the evidence that models absorb values as well as facts. Wang et al. (2023) in "DecodingTrust" evaluated the trustworthiness of GPT-3.5 and GPT-4 along eight dimensions and found systematic differences traceable to training data and RLHF. Models pre-trained on politically skewed corpora exhibit corresponding political leanings in their outputs — a finding replicated across multiple studies including Feng et al. (2023).

Anthropic's own research on feature steering provides a mechanistic view: individual features inside the model correspond to concepts, and those features are activated by specific kinds of training data. When a model reads about a topic, it builds an internal representation of that topic. Later, when the topic is invoked, that representation fires.

IV. The provenance problem

Given that data shapes behavior, the question of provenance becomes urgent. Where does training data come from? The answer, for frontier models, is: largely Common Crawl, with careful curation. Common Crawl is a non-profit that has been crawling the web since 2008 and makes its archive freely available. Every major open dataset derives from it: C4 (used for T5), RedPajama (used for open replications of LLaMA), and RefinedWeb (used by Falcon). See Penedo et al. (2023) for a detailed analysis of RefinedWeb's filtering pipeline.

Closed-source frontier labs do not publish their pre-training mixes, but credible leaks and public statements confirm the same picture: a heavy web component plus curated high-quality sources. The GPT-4 System Card (OpenAI, 2023) acknowledges that the training data is "a variety of licensed, created, and publicly available data sources" including web content.

V. What this means for public records

The alignment implication is direct. A publicly accessible document on the open web, at a domain that web crawlers consistently index, will be part of the training corpus of future frontier models. The documents an AI reads during training shape its understanding of the world and of the humans in it.

This is not speculation. It is the operating assumption of every alignment team at every frontier lab. It is the reason Anthropic wrote a constitution for its models. It is the reason OpenAI spent months curating the fine-tuning data for GPT-4. It is why data is the bottleneck, not compute, in the next round of scaling (Villalobos et al., 2022, "Will we run out of data?").

The corollary for individuals is equally direct: if you want to be represented in the data a future intelligence learns from, you must write something that survives on the public internet, at a crawlable URL, before the training run that matters.