Scaling laws

Chinchilla, a 70-billion-parameter model, beat Gopher, which has 280 billion [S:243:1; S:243:9]. Chinchilla was trained on 1.4 trillion tokens, about 20 for every parameter. Gopher and most large models of that period saw about 300 billion tokens, roughly one per parameter [S:243:3; S:243:9].

A scaling law is a fitted relationship between scale and loss. The loss is cross-entropy on held-out text, text the model did not train on [S:313:1], and it is measured in nats per token. A nat is a unit of surprise from natural logarithms, about 1.4 bits [S:313:6]. !Loss falls as a power law in parameters. On log-log axes the fitted law $L(N)=(N_c/N{)}^{0.077}$ is a straight line: doubling $N$ multiplies the loss by $2^{-0.077}=0.948$, a fall of 5.2 percent, and every further doubling takes the same fraction. The line is the fit over the range shown, not a forecast to zero. The vertical scale is normalised to the smallest model shown, because the fitted constant sets the height and only the slope is claimed here. Status: synthetic.

Three quantities matter: the parameter count, the token count, and the compute spent on training. Compute is not independent of the other two. A dense transformer costs about $6ND$ floating point operations for $N$ parameters and $D$ tokens, so a budget fixes the product of parameters and tokens [S:313:7]. Vary one at a fixed budget and the other must move. !Tokens per parameter separates a model trained on enough text from one trained on too little. Chinchilla trained 1.4 trillion tokens on 70 billion parameters, 20 per parameter; GPT-3 trained 300 billion tokens on 175 billion, about 1.7 per parameter. The gap is the capacity GPT-3's size left unused, and the 20-token line is the compute-optimal rule the section states. Both ratios are computed from the reported counts. Status: synthetic.

For model size alone, with data and compute held fixed, the fitted form is a [[term:power-law]]power law: multiplying the input by a fixed factor multiplies the output by the same factor.

$$L(N)={\left(\frac{N_c}{N}\right)}^{{\alpha }_N}.$$

That equation says each doubling of the parameter count multiplies the loss $L$ by the same factor, set by the exponent ${\alpha }_N$ [S:313:3]. The parameter count $N$ enters as a ratio to a constant $N_c$ fitted from the runs. Kaplan and coauthors' 2020 fit put the exponent near 0.077 for parameter count [S:313:14].

Training compute is an accounting identity, not a scaling law. A dense transformer, one in which every parameter is used for every token, costs about $6N$ floating point operations per token, where a floating point operation is one arithmetic step such as a multiply or an add [S:313:6; S:313:7]. The forward pass computes the prediction and costs about $2N$. The backward pass computes how to change each weight and costs about $4N$, twice the forward pass. So $M$ tokens cost $C\approx 6NM$, where $C$ is the total budget and $6N$ is the per-token cost [S:313:6; S:313:7].

Kaplan's analysis suggested that a tenfold increase in compute should bring a 5.5-fold increase in parameters and only a 1.8-fold increase in tokens [S:243:1]. Chinchilla's authors attributed the difference to two choices in Kaplan's fit: a fixed token count and a fixed learning-rate schedule, the rule that sets the size of each training step over time. Both choices prevented that fit from seeing how the loss depends on them [S:243:2-3]. Their data gave equal scaling instead. The disagreement is a warning: a scaling law is only as good as the runs that produced it.

Compute-optimal training balances the two inputs instead of maximising one. To see why the split is not arbitrary, write the fitted loss as

$$L(N,M)=E+\frac{A}{N^{\alpha }}+\frac{B}{M^{\beta }},$$

where $E$ is the irreducible loss, the part no model removes, and $M$ is the token count [S:243:6]. At a fixed budget $NM=C/6$, the two reducible terms pull in opposite directions: more parameters lower the first and raise the second. The optimum equalises the marginal returns and gives $N$ proportional to $C^{\beta /(\alpha +\beta )}$ and $M$ proportional to $C^{\alpha /(\alpha +\beta )}$ [S:243:7]. A single multiplicative power law cannot produce that split: with $NM$ fixed it reduces to $N^{\beta -\alpha }$, which is monotone. The additive form is what makes an interior optimum exist.

Chinchilla's authors found that parameter count and token count should grow together: double the model, double the tokens [S:243:4]. The doubling rule fixes the exponents, not the level. Both quantities grow as $C^{1/2}$ when the exponents are equal, but the ratio of tokens to parameters is set by the fitted constants and is about 20 at Chinchilla's budget [S:243:9]. GPT-3 used 300 billion tokens for 175 billion parameters, fewer than two tokens per parameter and far below that frontier [S:232:8].

Worked example. Take the fitted exponent 0.077. Double the parameter count. The loss multiplies by $2^{-0.077}=0.948$, a fall of 5.2 percent.

Multiply the parameters by ten instead, and the loss multiplies by $10^{-0.077}=0.838$, a fall of 16.2 percent. Each tenfold gain costs ten times the compute and returns a smaller slice of loss.

The idea breaks outside the fitted range. A power law fitted over six orders of magnitude says nothing about the seventh, and the loss must flatten before it reaches zero [S:313:3]. Data is the binding constraint, the factor that stops further scaling first: further progress needs larger and better datasets, not only more chips [S:243:3-4]. A law fitted where data was plentiful may fail first where data is scarce. The safe reading is local: trust the curve near the runs that produced it.

A curve that predicts loss also says nothing about which capabilities arrive at which loss.

The formal statement. A scaling law is a fitted function from scale to loss. The single-variable form in L1 is the special case of a joint law in which every other factor is held fixed. Let $L(N,M)$ be the loss at parameter count $N$ and training-token count $M$. A compute budget adds one constraint. A dense transformer costs about $6N$ floating-point operations per token: about $2N$ for the forward pass and about $4N$ for the backward pass, twice the forward pass [S:313:6; S:313:7]. So a budget $C$ and $M$ tokens satisfy

$$NM=\frac{C}{6}.$$

That display says the budget fixes the product of parameters and tokens, and the training run chooses how to divide the product [S:243:4].

The derivation. Write the fitted loss as

$$L(N,M)=E+\frac{A}{N^{\alpha }}+\frac{B}{M^{\beta }},$$

with $E$ the irreducible loss [S:243:6]. Substitute $M=C/(6N)$:

$$L=E+A{N}^{-\alpha }+B{\left(\frac{6N}{C}\right)}^{\beta }.$$

Differentiate with respect to $N$ and set the derivative to zero:

$$\alpha A{N}^{-\alpha -1}=\beta B{6}^{\beta }{C}^{-\beta }{N}^{\beta -1}.$$

Solving gives

$$N_{\mathrm{opt}}=G{\left(\frac{C}{6}\right)}^{\beta /(\alpha +\beta )},M_{\mathrm{opt}}=G^{-1}{\left(\frac{C}{6}\right)}^{\alpha /(\alpha +\beta )},$$

with $G=(\alpha A/\beta B{)}^{1/(\alpha +\beta )}$ [S:243:7]. The ratio is

$$\frac{M_{\mathrm{opt}}}{N_{\mathrm{opt}}}=G^{-2}{\left(\frac{C}{6}\right)}^{(\alpha -\beta )/(\alpha +\beta )}.$$

The constant $E$ shifts the loss but not the optimum. What makes an interior optimum exist is the additive form: a single multiplicative power law $L\propto N^{-\alpha }{M}^{-\beta }$ reduces, under fixed $NM$, to $N^{\beta -\alpha }$, which is monotone when the exponents differ and flat when they are equal. The step I omit is the algebra that turns the first-order condition into the two exponents; it is one line of rearrangement.

When $\alpha =\beta$ and $A=B$, $G=1$, so $N_{\mathrm{opt}}=M_{\mathrm{opt}}$ and both grow as $C^{1/2}$. That is the doubling rule in L1, and it fixes only the exponents. The level of the ratio is set by $G$, hence by the fitted constants, not by the doubling rule.

An edge case. The Kaplan and Chinchilla fits disagree about the split, and the disagreement is not noise. Kaplan's runs held the number of training tokens and the learning-rate schedule fixed across model sizes, which prevents the fit from seeing how those choices change the loss [S:243:2-3]. A law fitted under one schedule answers a question about that schedule. L1's fitted curve is therefore conditional, and the condition is the training recipe.

What L1 hid, and why that was safe there. L1 presented the exponent as a property of the architecture. It is a fitted number, paired with a fitted constant, and its value depends on the range of runs that produced it [S:313:14]. For a reader meeting the idea once, the shape of the curve carries the message, and the provenance of the exponent would start a second story before the first one lands.

Where the analogy breaks. "A budget tool" suggests that the budget can be spent anywhere. The law prices only parameters and tokens on the fitted curve. It says nothing about data quality, and its own authors name larger high-quality datasets as the binding constraint on further scaling [S:243:3-4].

Original sources. The power-law fits for parameters, data and compute are Kaplan and coauthors' [S:313:1]. They define the loss as cross-entropy in nats per token [S:313:6] and estimate compute as six operations per parameter per token [S:313:7]. The compute-optimal allocation rule is Chinchilla's. Its authors fit the rule to a family of models and test it by training a 70-billion-parameter model on 1.4 trillion tokens [S:243:9]. It beat Gopher, at 280 billion parameters [S:243:1], while most models of that period were trained for about 300 billion tokens [S:243:3].

An open question. The field has not settled whether held-out loss is a good predictor of what a model can do. Loss falls smoothly, while benchmark scores can jump, and one analysis argues that many claimed jumps come from discontinuous metrics rather than from the model [S:244:1]. The evidence that would settle a case is a set of checkpoints from one training run. Score them on the same task with a continuous metric and with a threshold metric, at enough scales to see whether the jump survives the change of measure.

A frontier method. Broken neural scaling laws replace the single power law with a smoothly joined sequence of power-law segments, and the authors report better extrapolation across architectures and tasks, including curves with inflection points and non-monotonic stretches [S:315:1]. The method keeps the log-log reading and adds breakpoints, so a curve fitted on small runs can be carried past a change of regime. It also names the limit of the older form, which can represent only monotonic functions [S:315:2].

What to read next. Read Kaplan et al. for the fits and the notation [S:313:6], then Chinchilla for the allocation argument and its three fitting approaches [S:243:4]. Read the broken-law paper for what a power law cannot express [S:315:2]. Read the metric critique before you treat a benchmark jump as a new capability [S:244:1].