The bias–variance trade

Eighty observations, each with 20 predictors, the recorded inputs the rule compares, and a target, the number you want to predict. A nearest-neighbour rule predicts a new case's target by averaging the targets of its k closest training cases; k is the number of neighbours, and the training data are the cases it may look at.

With one neighbour the prediction follows every wobble in the training data, while with many neighbours it smooths them away. The test error, the error on cases the rule did not train on, falls as k grows, reaches a minimum at about five neighbours in the source's regression example, and then rises [S:64:245], so neither end is best. !Estimating a mean of 1 from n observations by reporting c times the sample average, so c = 1 is the unbiased rule and c = 0 ignores the data. In (a), with ten observations, the squared bias falls and the variance rises as c grows toward 1, and their sum bottoms out at c = 10/11 with error 0.091, below the 0.1 of the unbiased rule. The best rule is wrong on average and still wins. In (b) the same sum is drawn for four sample sizes: the marked minimum sits at c = n/(n+1), so it moves toward 1 as the data accumulate, and the whole curve drops. More data does not remove the case for shrinkage; it shrinks the amount worth taking. Status: re-simulated.

The reason is that a slightly wrong model can trade a little bias for a large drop in variance.

The expected prediction error at a new input splits into three parts: the first is the noise in the target, which no model can remove. The second is the squared bias of the fitted model, the gap between its average prediction and the truth. The third is the variance of the fit, how much the prediction moves when the training set changes [S:64:242], and the split is exact for squared error.

The formula says the error you will see is irreducible noise plus a systematic gap plus a scatter term [S:64:242], and only the last two are under your control.

The noise term sets a floor that no model can go below, since even the true function has that error if the target varies for reasons the inputs do not record. A model that reports zero training error has fitted the noise rather than removed it, so the best possible test error is the noise, and a useful model gets close to it.

A complex model has low bias and high variance, because it can bend to fit the truth and also bends to fit the noise, so a new training set produces a different fit.

A simple model has high bias and low variance, because it ignores the noise and also ignores real structure. As complexity grows, the squared bias falls and the variance rises [S:64:242], so the best model sits where their sum is smallest, not where either part is smallest.

The two failures have names: fitting the noise is overfitting, and it shows up as low training error and high test error. Ignoring real structure is underfitting, and it shows up as high error on both, so the trade is not a choice between good and bad models but between two ways of being wrong.

Training error falls as complexity grows, because a flexible model can always fit the data it has seen, but prediction error does not. The gap between the two is the part of the error caused by fitting noise, and it grows when the model is allowed to memorize, which is why a model cannot be judged on the data that trained it.

Here is the arithmetic on two candidates: model A is unbiased and has variance 9, so its MSE is 0 + 9 = 9, while model B has bias 2 and variance 4, so its MSE is 4 + 4 = 8. Model B is wrong on average and still predicts better.

If the irreducible noise is 4 for both, the totals are 13 and 12, and the ranking does not change, because a slightly wrong model wins when the variance it saves exceeds the square of the bias it adds [S:64:245].

The nearest-neighbour example is the same trade in disguise, since small $k$ gives a flexible fit with low bias and high variance, while large k gives a stiff fit with high bias and low variance. As k grows the variance of the fit falls and its squared bias rises, and the sum is smallest in the middle of the range [S:64:245], so the minimum is neither the most flexible model nor the smoothest one.

More data shifts the balance, but only for some rules. For a fit with a fixed number $p$ of parameters, the variance falls like $p{\sigma }^2/N$ while the bias stays where it was, so the best complexity grows with the training-set size $N$ [S:64:243]. For a fixed-k nearest-neighbour rule the variance is ${\sigma }^2/k$ and does not fall as $N$ grows; there the variance falls only if k itself grows with $N$ [S:64:242]. This is why the best complexity is a property of the sample size as well as of the problem, and why a fixed rule for choosing it cannot exist.

The trade also explains why a model can improve on the training set while getting worse on new data. Adding a predictor lowers training error, because the fit has one more way to accommodate the sample, but it raises variance, because the fitted coefficient is estimated from noise as well as signal. The training curve is therefore a bad guide to the test curve, and the gap widens with complexity.

The decomposition is exact only for squared error. Under 0-1 loss, the score that counts a wrong label as 1 and a correct one as 0, prediction error is no longer the sum of squared bias and variance, and the curves can behave differently [S:64:245]. This is the setting of classification, where the target is a category rather than a number, and bias and variance are also properties of the fitting procedure over repeated training sets, not of the one fit you have. You cannot read them off a single dataset without a model of how the data vary, which is why the next chapter turns to validation.

The formal statement. Let $Y=f(X)+\epsilon$ with $E[\epsilon \mid X]=0$ and $\mathrm{Var}(\epsilon \mid X)={\sigma }^2$. Let $\hat{f}$ be fitted on a random training set, and evaluate it at a fixed input $x_0$. Under squared loss, the expected error over training sets and over a fresh $Y$ is

The three terms are irreducible noise, squared bias and variance [S:64:242]. The identity needs squared loss, an independent test point, and additive noise whose mean is zero and whose variance does not depend on the input.

The derivation. Add and subtract the two means:

The first term is the noise. Square the sum and take expectations. The cross term with the noise vanishes because the noise has mean zero and is independent of the fit. The cross term between the two centered fit terms vanishes because a deviation from its own mean has mean zero. What remains is the three terms. No step is omitted for the squared-error case.

The edge case. Under 0-1 loss the three terms do not add, and prediction error can fall while squared bias rises [S:64:245]. A classifier can sit on the correct side of the boundary everywhere and still have large squared bias, because only the sign of the fitted probability matters. L1's arithmetic is safe because it uses squared error. A reader who carries "bias squared plus variance" into classification gets the wrong curve, and the wrong location for the best complexity.

What L1 hid. "Complexity" is not one number. The U-curve assumes a one-dimensional family ordered by flexibility, such as the number of neighbours $k$. Modern models add a second descent beyond the interpolation threshold, where more capacity lowers test error again [S:309:1]. L1's nearest-neighbour example is safe because $k$ orders the models along one axis. The decomposition also concerns the expected error over training sets, not the error of the one fit you have.

Where the analogy breaks. The trade suggests a single bottom, and the U-curve is drawn that way. Double descent shows a second regime in which larger models do better [S:309:1]. The analogy also treats the noise floor as fixed, while in real data the noise and the model's misspecification are hard to separate.

Original sources. The decomposition is equation 7.9 of The Elements of Statistical Learning, derived under squared loss with additive noise [S:64:242]. L1's arithmetic copies the estimator version of the same identity, not the prediction version. The prediction version is the one that names irreducible noise, and it is the version that the nearest-neighbour example uses.

An open question. The mechanism behind model-wise double descent in deep networks remains an open question [S:311:5]. It is also open whether more data can hurt when early stopping is optimal [S:311:10]. Evidence that would settle a given case: an experiment that varies capacity and sample size jointly under a prespecified stopping rule, or a theory that predicts the location of the peak from measurable properties of the model and the data. The peak also shifts when the training procedure changes, which is why a one-dimensional complexity axis is not enough [S:311:10].

A frontier result. Belkin and colleagues show that increasing capacity past the interpolation threshold lowers test risk again, and call the curve double descent [S:309:1]. Nakkiran and colleagues extend the phenomenon to training time and to sample size [S:311:4]. Both results keep the three-term bookkeeping. They ask what happens when the variance term stops falling and the interpolating model is chosen by a different bias, such as the smallest norm among the fits [S:309:3].

What to read next. Read ESL 7.2 and 7.3 for the derivation, the nearest-neighbour example and the 0-1 edge case [S:64:242] [S:64:245]. Read Belkin and colleagues for the curve and its mechanism [S:309:1]. Read Nakkiran and colleagues for the experiments, the definition of effective model complexity, and the open questions [S:311:5] [S:311:10].