Bias, variance and MSE

Two scales weigh the same parcel. Scale A is correct on average, but its readings jump around with a variance of 9 square grams, the average squared size of a miss. Scale B reads 2 grams too heavy every time, and its readings vary with a variance of 4. You may use one scale for a single weighing, so which do you choose?

The answer is scale B, because its steady offset costs 4 units, the square of its 2-gram bias, and its scatter costs 4, for a total of 8. Scale A pays nothing for an offset but pays 9 for scatter alone. The rest of this section is the accounting. !Scale A is right on average and scatters with variance 9; scale B reads 2 grams heavy every time and scatters with variance 4. In (a) the bell curves show the two faults: A is centred on the truth and wide, B is narrow and off centre. In (b) the parts add up: A pays 0 + 9 = 9, B pays 2 squared + 4 = 8, so B has the lower mean squared error for a single weighing. The zero bias segment has no width and is written beside the bar. Averaging many readings reverses the ranking, because repetition shrinks the variance of the average and leaves the 2-gram offset in place. Status: schematic.

The error of a guess has two sources, and the first is bias: the expected difference between the estimator and the quantity it estimates [S:63:190]. Scale B has a bias of 2 grams and scale A has a bias of 0; a rule with zero bias is unbiased [S:63:190]. Bias is systematic: repeating the measurement does not remove it.

Bias is a property of the rule, not of the number you report, so you cannot look at one estimate and say whether it is biased, because the truth is unknown. You can only ask how the rule would behave across repeated experiments. This is why the chapter keeps returning to the sampling distribution: every property in this section is an average over data you did not collect.

The second source is the variance of the estimator, which measures how much the estimate changes as the data change [S:63:190]. Scale A has variance 9, so its readings scatter widely around the correct weight, while scale B has variance 4, so its readings cluster tightly around a wrong value. The units are square grams because a variance is an average of squared errors.

Repeating the measurement averages the scatter away, but it never averages the bias away, and that difference decides which scale is better for one weighing and which is better for many.

Do not confuse the variance of the estimator with the variance of the data. The data have their own spread, which is a property of the population, while the estimator has a spread across repeated samples that shrinks as the sample grows. A noisy population can still yield a precise average, provided the sample is large.

The two sources combine into one number, the mean squared error, which is the expected squared difference between the estimate and the truth, averaged over repeated samples. Write the estimate as $\hat{\theta }$, its average over samples as $m$, and the truth as $\theta$. Then $\hat{\theta }-\theta =(\hat{\theta }-m)+(m-\theta )$; squaring and averaging, the cross term vanishes because $\hat{\theta }-m$ averages to zero, and what remains is:

The formula says the average squared error is the scatter plus the systematic offset squared [S:63:191], so a bias of 2 costs 4 units of MSE, the same as a variance of 4.

The two numbers from the opening now have names: scale A's 9 is all variance and no bias, while scale B's 8 is 4 for variance and 4 for the square of its offset. The steadier scale wins for a single weighing, even though it is wrong on average.

For an average of $k$ weighings, scale A's mean squared error is $9/k$ and scale B's is $4/k+4$, because averaging divides the variance and leaves the bias alone. Scale A wins from the second weighing on, since at $k=2$ its error is 9/2 against B's 6, and the bias is punished as soon as the variance is averaged, not only after many weighings.

This is the first appearance of a pattern that runs through the chapter: a biased rule can beat an unbiased one when it buys enough reduction in variance [S:63:191]. The trade is not a loophole in the mathematics; it is what the mathematics says once you agree to score rules by squared error. Section 13.6 applies the same arithmetic to prediction rather than to a single quantity.

Take $n$ observations from a Gaussian, the bell curve, and consider the rule that divides the sum of squared deviations by n, whose expected value, an average over repeated samples, lies below the true variance [S:63:190]. It comes out at (n−1)/n of the truth, because the deviations are measured from the sample average and one of the n pieces of information is spent. With n = 4 that is three quarters, a shortfall of one quarter, while the rule that divides by n−1 is unbiased [S:63:190-191].

The two rules rank differently on squared error than on bias: for Gaussian data their mean squared errors are ${\sigma }^4(2n-1)/n^2$ and $2{\sigma }^4/(n-1)$, and the ratio is $(2n-1)(n-1)/(2n^2)$, below 1 for every $n>1$ and tending to 1. The ÷n rule therefore has the smaller mean squared error at every sample size, and the ÷(n−1) correction buys unbiasedness at a price in squared error. If you need an unbiased estimate, pay that price, but if you are scored on squared error the biased rule wins, and the gap narrows as the sample grows.

The decomposition is exact for squared error and for expectations taken over repeated samples. It is not a description of the single dataset in front of you, and you cannot read bias and variance off one sample without a model. Squared error also treats a 10 gram underestimate exactly like a 10 gram overestimate, so if one direction is worse, MSE is the wrong score, and the decomposition answers a question you did not ask.

The square in the formula is not decoration, because it makes positive and negative errors count the same and makes one large miss cost more than several small ones. That is a choice, not a law: if a two-gram error is four times as bad as a one-gram error, squared error fits; if it is only twice as bad, the absolute error is the honest score. The decomposition in this section belongs to the squared choice.

The formal statement. Let $T$ estimate a fixed $\theta$, and write $m=E[T]$ for its mean over repeated datasets. If $E[T^2]$ is finite, then

The first term measures scatter around the rule's own average. The second is the squared bias, the distance from that average to the truth [S:63:191]. The identity holds at each candidate truth separately, and it assumes squared loss and a finite second moment.

The derivation. Expand the square around $m$:

The right side is $E[(T-m{)}^2]+2(m-\theta )E[T-m]+(m-\theta {)}^2$. The middle term vanishes because $E[T-m]=0$. What remains is the variance plus the squared bias. No step is omitted, and the only hypothesis used is that the expectation exists.

The edge case. For the Gaussian variance rule the ranking does not reverse with the sample size. The estimator that divides by $n$ has bias $-{\sigma }^2/n$ [S:63:190], and $(n-1)s^2/{\sigma }^2$ has a ${\chi }^2$ distribution with $n-1$ degrees of freedom [S:135:594], whose variance is twice its degrees of freedom. Hence $\mathrm{Var}({\sigma }_{\mathrm{mle}}^2)=2{\sigma }^4(n-1)/n^2$ and $\mathrm{Var}(s^2)=2{\sigma }^4/(n-1)$, and adding the squared biases gives $\mathrm{MSE}({\sigma }_{\mathrm{mle}}^2)={\sigma }^4(2n-1)/n^2$ and $\mathrm{MSE}(s^2)=2{\sigma }^4/(n-1)$. Their ratio is $(2n-1)(n-1)/(2n^2)$, below 1 for every $n>1$ and tending to 1, so the biased rule has the smaller mean squared error at every sample size. Unbiasedness is a different criterion, and the correction is the price paid for it, not an improvement that arrives with large $n$.

What L1 hid. L1 wrote the bias as a fixed number of grams. In general the bias is a function of the unknown truth, so the decomposition holds pointwise and two rules can swap places as the truth moves. The scale story is safe because the offset is physical and constant. L1 also hid the finite-variance condition, which fails for heavy-tailed data.

Where the analogy breaks. A scale that reads heavy by two grams is wrong in a way no repetition removes. That part survives. The analogy breaks when it suggests that bias and variance are properties of one instrument. They are properties of a procedure over imagined repetitions, and the repetitions are not the ones you performed. It also breaks because a correction can trade bias for variance, so "wrong on average" and "worse for you" are different claims.

Original sources. Murphy defines the bias and the variance of an estimator and derives the identity that L1 uses [S:63:190] [S:63:191]. He attributes the tradeoff to an earlier reference [S:63:191]. The scale comparison in L1 is one instance of that identity, not a separate argument.

An open question. In three or more dimensions the maximum likelihood estimator is inadmissible, and the James–Stein estimator has lower risk at every value of the parameter [S:34:589]. There is a continuum of estimators that attain the minimax risk, so minimaxity alone does not choose one [S:34:589]. The field has not settled which selection principle to use when the parameter has unknown structure. Evidence that would settle a given case: a criterion that selects one estimator with a finite-sample guarantee over a stated class of parameters, checked against the minimax risk on that class.

A frontier result. The James–Stein estimator shrinks the observation toward zero by a factor that depends on the data [S:34:586]. It is a clean case where a biased rule beats an unbiased one, and the same shrinkage idea appears in modern regression. L1's two-scale example is the same arithmetic with one parameter. Stein's result shows that the gain appears when you estimate many parameters at once. The gain is stated for squared error, which is why L1's warning about the choice of loss still applies.

What to read next. Read Murphy 4.7.6 for the derivation and the Gaussian shrinkage example, where a biased MAP estimate beats the unbiased mean [S:63:191] [S:63:192]. Then read Polyanskiy and Wu chapter 28 for risk functions, minimaxity and the James–Stein estimator [S:34:586] [S:34:589]. For the high-dimensional setting, those two chapters are the entry point to the modern literature on shrinkage.