Pretraining

GPT-3, a 175-billion-parameter model released in 2020, was trained on 300 billion tokens of text [S:232:8]. Llama 3, released in 2024, was trained on about 15.6 trillion, roughly 52 times more [S:431:1]. The scale changed, not the basic task. In both runs, the model read raw text and was asked to predict what came next [S:91:13].

That is what pretraining means here: training a model on a broad body of text before any task is named [S:91:8]. The supervision is self-supervised, because the label for each position, the answer the model is scored against, is the text that is already there [S:91:10]. The signal that says what the right answer was comes from the data, not from a person. !One sentence, three self-supervised objectives. A decoder-only model predicts each token from the tokens on its left; BERT masks 15 percent of the input and predicts those tokens from both sides; T5 replaces spans with sentinel tokens and reconstructs them. In all three, the label is text that is already there. The figure is schematic. Status: schematic.

Two objectives shaped pretraining. A decoder-only model such as GPT reads left to right and predicts the next token, with no separate encoder [S:220:3]. BERT, an encoder-only model from 2018, selects 15 percent of token positions and predicts the original token at each one from both sides [S:219:4]. Of those selected positions, 80 percent become a mask symbol, 10 percent a random token, and 10 percent stay unchanged [S:219:4]. The bidirectional view suits classification and extraction, where the whole input is available, but it does not write text one token at a time [S:219:1]. !Surprise is not shared evenly across tokens. The four correct-token probabilities 0.5, 0.25, 0.8 and 0.1 cost 1, 2, 0.322 and 3.322 bits, so the confident mistake at 0.1 dominates the average of 1.661 bits. Exponentiating gives a perplexity of 3.16, against 4 for a uniform choice among four tokens. The four probabilities are synthetic, taken from the section's worked example. Status: synthetic.

Nothing in either objective mentions translation, arithmetic or question answering. The bet is that the global minimum of the next-token objective is also a minimum for those tasks, because the demonstrations are in the text [S:224:3]. That is a statement about the optimum, not about a trained model: whether training reaches it is an empirical question, so a real model may represent only part of the regularities. The objective is the same at every size, which is what makes scale comparisons meaningful.

The loss makes the objective precise. For a correct token with predicted probability $p$, the surprise is $-\log p$. The logarithm answers "what power do I raise the base to"; in base two the surprise is measured in bits, one bit for each halving of the probability. The training loss $H$ is the average surprise per token:

$$H=-\frac{1}{T}\sum _{t=1}^T\log P(w_t\mid w_1,\dots ,w_{t-1}).$$

The summation sign adds the surprises of all $T$ positions, and dividing by $T$ turns the total into an average. The model is rewarded for putting high probability on the token that occurred, and the logarithm makes a confident mistake expensive [S:220:3]. The same quantity is called cross-entropy when it is read as an average code length, the number of bits a code would need per token.

Exponentiated in the same base, it gives perplexity: $2^H$ when $H$ is in bits and $e^H$ when it is in nats. Perplexity is the effective number of equally likely tokens among which the model is choosing. It is not a probability; it is a count of effective alternatives.

Worked example. Four predictions assign probabilities 0.5, 0.25, 0.8 and 0.1 to the tokens that actually occurred. In bits, the surprises are 1, 2, 0.322 and 3.322, because ${\log }_{2}0.8=-0.322$ and ${\log }_{2}0.1=-3.322$. Their average is 1.661 bits per token.

Perplexity is $2^{1.661}=3.16$, so the model behaves like a uniform choice among about three tokens. A uniform model over four tokens would score 4. Perplexity lets you compare runs on the same tokeniser, because it removes the length of the test text from the score [S:98:54]. The average is taken over every token, so one very unlikely token can dominate a short run.

The idea has two costs. The first is data. Web text is not automatically good text: the C4 crawl, a cleaned collection of web pages built for T5, still needed filtering for boilerplate, code and duplication, and produced about 750 GB [S:221:7]. A training corpus is the body of text a model reads. GPT-3 mixed filtered web pages, books and Wikipedia, and weighted the mix by hand rather than by raw size [S:232:9-10].

The second cost is measurement. Perplexity averages over tokens, so models with different tokenisers are not measured in the same unit [S:98:54]. A model can also memorise test text that leaked into its training corpus, which flatters every score computed on that text [S:232:9].

The formal statement. Let a corpus be a token sequence $w_1,\dots ,w_T$, and let the model assign each token a conditional probability given the tokens before it. Pretraining selects parameters that make the corpus likely. The chain rule writes the corpus probability as a product, and the logarithm turns that product into a sum: $$\log P(w_1,\dots ,w_T)=\sum _{t=1}^T\log P(w_t\mid w_1,\dots ,w_{t-1}).$$ The training loss is the negative of that sum [S:220:3].

Dividing by $T$ does not change which parameters win, because $T$ is fixed by the corpus. It changes the unit, from total surprise to average surprise per token [S:98:54].

The derivation. Each conditional probability is at most one, so the likelihood of a long sequence is small and multiplying many of them loses precision. The logarithm is monotone, so maximising the product and maximising its logarithm choose the same parameters. The logarithm of a product is the sum of the logarithms, which gives the display. Negating turns maximisation into minimisation, and dividing by $T$ gives a number that does not grow with corpus length.

One step is omitted: the proof that a softmax layer can represent any distribution over the vocabulary. That is a claim about the final layer, not about the objective.

An edge case. L1 reads perplexity as the effective number of equally likely choices. That reading is exact only when the distribution is uniform over that many tokens. For a skewed distribution, perplexity equals a weighted average branching factor, not a count of live options [S:98:55]. A model can report perplexity 3 while placing most of its mass on a single token.

The phrase "about three tokens" then describes an average over positions, not the set the model actually chooses from.

What L1 hid, and why that was safe there. L1 treated the three pretraining objectives as variations on one loss. They are not the same likelihood. A masked objective predicts a token from both sides, so it does not define a probability for the sequence and has no perplexity in the sense above [S:219:1]. At a first pass the shared shape, predict something from the text, is the part that matters, and the difference is a second story.

Where the analogy breaks. "The label is the text that is already there" suggests that the answer is always present in the input. In masked modelling the answer is hidden at ninety percent of the selected positions, not all of them: BERT replaces eighty percent with a mask symbol and ten percent with a random token, and leaves the remaining ten percent unchanged [S:219:4]. The model still predicts the original token at every selected position. Self-supervision means that the supervision comes from the text, not that the answer stays visible.

Original sources. The decoder objective is GPT-1's likelihood over a window of preceding tokens [S:220:3]. The masked objective is BERT's: it selects 15 percent of the token positions, masks 80 percent of the selected positions, and predicts the original token at each one from both sides [S:219:4]. The span-denoising objective is T5's, with sentinel tokens marking the removed spans [S:221:21]. T5's own sweep found the encoder-decoder with denoising best across its tasks [S:221:19].

An open question. The field has not settled how much of a pretrained model's quality comes from the objective and how much from the corpus. The two usually change together, so a gain cannot be assigned to either. FineWeb's authors document and ablate the curation choices, including filtering and deduplication strategies, and release the models trained in those ablations [S:430:1]. That is the shape of the evidence that would settle a comparison: one tokeniser, one compute budget, one evaluation, and one change at a time.

A frontier method. FineWeb is a 15-trillion-token dataset built from 96 Common Crawl snapshots, and FineWeb-Edu is a 1.3-trillion-token subset filtered by a classifier for educational value [S:430:1]. Models trained on the educational subset score better on knowledge- and reasoning-heavy benchmarks [S:430:2]. The method depends on this section because the filter is trained on text and the payoff is measured as a change in downstream scores. Curation became a modelling decision rather than a preprocessing detail.

What to read next. Read the T5 data section for what a cleaned crawl looks like and how large it is [S:221:7]. Then read GPT-3's dataset table for how a mixture is weighted by hand rather than by size [S:232:9-10]. Read FineWeb for the ablation method [S:430:1], and read the contamination paragraph in Speech and Language Processing before you trust any perplexity number [S:98:53].