Residuals and the first overfit

Four readings arrive from an instrument: 0 at time 0, 1 at time 1, 0 at time 2, and 1 at time 3. A straight line through them cannot pass through all four points, so it leaves a residual on every row. The least-squares line is $0.2+0.2x$, and it misses each row by a small amount. A curve with four adjustable coefficients can pass through all four exactly, and its residual sum of squares is zero. By the measure of Section 8.4 the curve wins without a contest, and that verdict is the first trap of modelling.

With as many coefficients as rows, a polynomial can be made to hit every point, since four rows give four conditions and four coefficients are enough to meet them. A residual of zero is therefore a property of the construction rather than evidence about the world. MML, Murphy's Machine Learning: A Probabilistic Perspective, describes the result as a fit that is too good on the training data — the rows used to choose the coefficients. Such a fit does not generalize to test data, the rows held back and used only to score the fit [S:27:296]. !Four readings and two fits. The line 0.2+0.2x misses each training row by a small amount and misses the held-back reading at 0.5 by 0.1. The cubic passes through all four training rows and misses the same reading by 0.6, so its squared error is 36 times larger on the row it never saw. Status: re-simulated.

The cubic through these four points is $0.667x^3-3x^2+3.333x$. It agrees with the data at the four recorded times and nowhere in between, returning 1.0 at time 0.5 where the straight line returns 0.3, and 0 at time 2.5 where the line returns 0.7, so the discrepancy is concentrated between the observations. !Residuals against time. The line leaves −0.2, 0.6, −0.6 and 0.2, a visible pattern; the cubic leaves four zeros because four coefficients met four conditions. A plot of exactly zero residuals marks a fit that memorised its rows, and neither panel alone says which model will predict a new one. Status: re-simulated.

Now keep one reading back. Suppose the instrument also reports 0.4 at time 0.5, and you did not use that row when fitting. The line is off by 0.1 and its squared error is 0.01, while the cubic is off by 0.6 and its squared error is 0.36. The model that fit the training rows worse predicts the held-back row better, by a factor of 36 in squared error. Overfitting is the name for this: the fit is excellent on the rows you hold and poor on rows you do not [S:27:296].

This is the first appearance of a pattern that returns throughout the book. A score computed on the rows used for fitting is optimistic, because the fit was chosen to make that score small. The optimism grows with the number of coefficients, and with four rows it can swallow whatever signal the data contained.

Generalisation is the word for performance on rows the model has not seen, and it is the quantity that the fit cannot measure about itself. The distinction between the two scores is the content of the next few chapters, and it is worth naming now. Training error is what the optimiser sees — the procedure that searches for the coefficients, here the least-squares solve — and test error is what the world sees.

The residual list is the diagnostic. For the line the residuals are −0.2, 0.6, −0.6 and 0.2, a pattern that alternates in sign, and for the cubic they are all zero by construction. Murphy describes a residual plot as the residuals drawn against the input, and says a reasonable model leaves a cloud around zero with no obvious trend [S:63:410]. A trend means the model has missed structure that is still visible in the errors, and a plot of exactly zero residuals is the signature of memorisation rather than of understanding. With four rows the plot cannot settle the question on its own: a trend the model missed and noise it has memorised can both leave a tidy cloud when the points are few and no row is held back.

Polynomial terms do not make the problem nonlinear. A curve in the input is still a linear combination of columns, each column multiplied by a coefficient and the results added, so the normal equations apply unchanged with one column per power [S:49:107]. That is why the trap is easy to fall into: the same solver that fits a line will fit a degree-nine curve, and it will report a cost that never rises when you add a column. The cost strictly falls only while the new column adds a direction the existing columns do not already span; once the columns span the rows, the cost is zero and stays there. The solver is doing exactly what it was asked to do, and the request was incomplete rather than the arithmetic wrong.

Interpolation means passing through every recorded row; regression means fitting a pattern that need not pass through any of them exactly. Interpolation is a guarantee about the recorded rows, not a statement about the underlying relationship, and confusing the two is the commonest mistake when regression is first used.

Holding rows back shrinks the training set, which makes the fit worse for a reason that has nothing to do with the model. A held-back score is itself one noisy number, since changing which row you keep can flip the ranking. That is why later chapters replace the single split with cross-validation: it repeats the hold-out, training on all but one part of the rows and scoring on the part left out, then rotates which part is left out. Resampling — drawing new training sets from the rows you have, by shuffling or by drawing with replacement — makes the estimate less dependent on an arbitrary division, but it cannot manufacture information that the four rows never contained.

The formal statement. Suppose each reading is a true value plus noise. The noise has mean zero and variance ${\sigma }^2$, and it is independent of the data used for fitting. Fit a model by least squares and let $\hat{y}(x_0)$ be its prediction at a fixed input $x_0$. Then the expected squared error at that input is

$$\text{expected error}={\sigma }^2+(\text{bias}{)}^2+\text{variance},$$

where the bias is the difference between the average prediction and the true value, and the variance is the spread of the prediction around its own average [S:64:242]. The expectation runs over the noise and over the draw of the training set.

The derivation. Add and subtract the average prediction. The error then splits into three pieces: the noise, the average prediction minus the true value, and the prediction minus its own average. Squaring produces three squared terms and three cross terms. Every cross term vanishes. The noise is independent of the fit and has mean zero, which kills the terms that contain it. The remaining cross term multiplies a deviation from its own mean by a constant, so it also has mean zero. What survives is the three-term identity. The hypothesis doing the work is independence of the noise from the training draw, and no step is approximate.

The edge case. The identity is pointwise at $x_0$ and specific to squared loss. With absolute loss there is no three-term split of this form. It is also an average over training sets that you never see. You hold one fit, so you cannot read the bias and the variance off it. That is why the decomposition explains the shape of a curve without measuring it. The empirical substitute is the gap between training error and test error, and that gap is itself a random quantity.

What L1 hid. L1 said the optimism grows with the number of coefficients. The sentence is right while the model cannot interpolate, and it stops being the whole story at the threshold. Once the coefficients reach the number of rows, the training error is zero by construction, and the test error can fall again as the model grows [S:49:441]. The classical curve is a U, not the whole curve. L1's four-row example sits exactly at the threshold, which is why the trap appears there and why the simplification was safe for that example.

Where the analogy breaks. L1's line that training error is what the optimiser sees and test error is what the world sees suggests the second number is a property of the world. It is an estimate from a finite sample, with its own variance. Change which row you hold back and the ranking can flip, so a test score is a measurement with an error bar, not a verdict.

Original sources. The decomposition appears as equation 7.9 of the ESL, with the irreducible error, the squared bias and the variance separated [S:64:242]. Murphy states overfitting as fitting the training data too well and failing to generalise [S:27:296], and describes the residual plot as the diagnostic for structure the model missed [S:63:410]. The ESL adds the warning that training error falls to zero as complexity grows, while a model with zero training error typically generalises poorly [S:64:240].

An open question. The field has not settled why interpolation sometimes generalises. Double descent names the pattern: test error rises near the interpolation threshold and then falls again as the model grows [S:311:1]. The peak is real in the experiments, and the explanation in terms of effective model complexity is a hypothesis rather than a theorem [S:311:3]. The same paper states a sharper question: whether more training data can ever hurt when early stopping is tuned well [S:311:10]. Evidence that would settle it: a task where adding rows raises test error under optimal early stopping, reproduced across seeds, with the peak located by a complexity measure fixed before the experiment.

A frontier method. The minimum-norm interpolating fit is what makes the second descent possible: among the many fits with zero training error, the smooth one has lower variance [S:49:441]. That is the same minimum-norm idea as the pseudoinverse in Section 8.5, now applied to a model class rather than to one linear system. It connects the residual plot of this section to the implicit-regularisation results of the research literature.

What to read next. Read the ESL's chapter 7 [S:64:242] for the decomposition and [S:64:240] for the optimism of the training error, Murphy's chapter 11 [S:63:410] for the residual plot, and the ISLP's discussion of double descent [S:49:441]. Then read the empirical study in [S:311:1] for what the classical curve leaves out, and the open question it states on [S:311:10].