Ridge and shrinkage

A laboratory keeps two thermometers in the same room and writes both readings into every row of its table. The two columns are equal on every row, so between them they carry one fact. Ask least squares for a separate slope on each column and the question has no unique answer: a weight of 2 on the first thermometer and 0 on the second fits exactly as well as 1 and 1, or 0 and 2.

The fit is not wrong, and it is not usable either. A coefficient of 2 beside a coefficient of 0 invites the reading that the first thermometer matters and the second does not, while the table says nothing of the kind. Ridge regression settles the ambiguity by charging for size: the fit pays a price proportional to how large the coefficients are. The charge buys stability at the price of a small bias, meaning a fit that is systematically off the least-squares answer, and that trade is the reason to shrink at all. !Ridge on sixteen synthetic afternoons from data/scripts/c14.s03.L1.py (seed 16); the two sunshine records differ only by sensor noise. Top: the slopes begin at 3.25 and −2.31 and bend toward zero as λ rises, without arriving, and the second crosses zero on the way. Bottom: error on new afternoons falls to 0.164 at λ ≈ 1.69 and climbs again, while the training error only rises from 0.098 to 4.03. Status: synthetic.

Ridge adds a penalty on size to the residual sum of squares, the total of squared misses that least squares minimises:

The penalty is small when the coefficients are small, so the fit pays for two things at once: matching the table and staying modest [S:49:248]. The sum runs over the coefficients j, and the tuning parameter λ, a number you choose, sets the exchange rate between those two aims. At λ = 0 the penalty vanishes and ridge returns the least-squares answer, while a growing λ pulls the coefficients toward zero [S:49:249].

The intercept is exempt from the charge, and the exemption is deliberate. Shrinking the intercept would drag every prediction toward zero regardless of what the table averages, while the quantity worth controlling is the association between each input and the answer [S:49:249]. Once the columns are centred, meaning each column has its mean subtracted, the fitted intercept is the mean of the recorded answers.

Shrinking costs a little bias and buys a large reduction in variance, the spread of the estimate across repeated tables. That trade is worth taking when the least-squares estimates are unstable, because a small change in the training table then produces a large change in the coefficients [S:49:250]. Instability is worst when the inputs are numerous or nearly duplicated, and when there are more columns than rows, least squares has no unique answer at all while ridge still returns one [S:49:251].

Least squares lands wherever the table pushes it, so an exact zero there would be an accident [S:49:238]. Shrinkage names the visible effect — coefficients sliding toward zero — not the arithmetic: they never arrive. That is why a zero has to come from a different penalty, not from a smaller ridge fit. Fitted values move toward the overall mean, so a column doing very little work loses most of its weight while a column carrying a clear signal keeps most of its own.

A single λ gives a single fitted model, but the useful object is the whole path. Sweep λ upward from zero and most coefficients trace curves that begin at their least-squares values and bend toward zero without ever arriving. An individual coefficient can occasionally rise as the charge grows, because the charge reshuffles weight between correlated columns [S:49:249]. Those curves answer a question one fit cannot: how much of the model survives when you insist on smaller coefficients, and where the useful compromise sits between a flexible fit and a stable one. A path also serves as a warning, because a coefficient that collapses under a small charge was probably resting on very little evidence in the first place.

Ridge needs one piece of housekeeping, because the penalty charges every coefficient the same amount whatever its column happens to measure. A column whose numbers are small carries a large coefficient, and the charge therefore punishes it harder than a column whose numbers are large, so the answer would depend on the units you happened to choose [S:49:250]. Standardizing each column, rescaling it so its standard deviation is one, removes that accident, at the price of one preprocessing step that costs a few lines of arithmetic. A model that mixes a salary in dollars with an age in years would otherwise spend its charge on the age, not because age matters more but because its column is recorded in smaller numbers.

Worked example. Take three rows with one input: x reads 1, 2 and 3, and y reads 2, 4 and 6. With a single copy of the column, least squares returns a slope of 2 and an intercept of 0.

Give the model two identical copies and, without a charge, their weights must sum to 2 so that the fitted line still has slope 2. With a charge the sum is free to fall below 2, and symmetry puts the same weight a on both copies, so the fitted line has slope 2a. Ridge with λ = 28 charges 28 for each squared weight. The squared inputs 1, 4 and 9 add to 14, so the residual sum of squares is 14 times the square of the quantity 2 minus 2a, the penalty is 56 times a squared, and the minimum sits at a = 0.5 for each copy.

Check the balance at that point. The predictions are 1, 2 and 3 against 2, 4 and 6, so the residuals are 1, 2 and 3, and their squares add to 14. The penalty is 28 times the sum of 0.25 and 0.25, which is also 14. The equality of the two totals is a coincidence of this charge: what the optimum equalises is the marginal loss in fit and the marginal rise in the penalty, not their levels. The even split is the price of an ambiguity the table cannot settle on its own.

The idea has a cost. Ridge never sets a coefficient to zero, so it cannot tell you which inputs matter [S:49:253]. It also hands you a fresh dial, λ, and the honest way to set that dial is to measure performance on data the fit never saw; a value chosen on the training table rewards the model for remembering it.

The formal statement. Centre the columns, so the intercept is the mean of the answers, and charge only the slopes. Ridge returns

$$\hat{\beta }(\lambda )={\left(X^{\mathsf{T}}X+\lambda I\right)}^{-1}{X}^{\mathsf{T}}y,$$

the minimiser of the residual sum of squares plus $\lambda$ times the squared length of the coefficient vector [S:49:248]. The single hypothesis is $\lambda \ge 0$. For $\lambda >0$ the matrix in brackets is positive definite whatever $X$ is, so the estimate exists even when the table holds more columns than rows and least squares has infinitely many zero-residual solutions [S:49:237] [S:49:251].

The derivation. Set the derivative of the penalised total to zero. What remains is a linear system, the least-squares system with $\lambda$ added to every diagonal entry. Adding a positive number to a diagonal cannot make an invertible matrix singular, and it repairs a singular one. Write the singular value decomposition $X=UD{V}^{\mathsf{T}}$. The fit then multiplies direction $j$ by the factor $d_j^2/(d_j^2+\lambda )$: a direction with a large singular value keeps most of its weight, and a direction with a small one loses most of it. Shrinkage is a tax on directions, not a tax on individual coefficients.

The edge case. L1 notes that one coefficient can occasionally rise as the charge grows, and the decomposition says why. Two correlated columns share a direction. Charging the pair shrinks the shared direction, and the split between the two coefficients then shifts, so one of them can grow even while the fitted values are pulled toward the mean [S:49:249-250]. If you hear "multiply each coefficient by a number below one", you will predict the opposite and be surprised by the path.

What L1 hid. L1 says the trade buys a large cut in variance for a little bias. The bias is not free-standing: it equals $-\lambda (X^{\mathsf{T}}X+\lambda I{)}^{-1}\beta$, which is zero at $\lambda =0$ and grows with the charge. The variance falls at the same time, so the mean squared error has a minimum that may sit inside the range of the charge or at one of its ends. When every true coefficient is zero the bias is zero throughout and the mean squared error falls all the way to the largest charge. The interior minimum the source reports belongs to its simulated table, not to ridge in general, and whether that minimum beats least squares is a property of the table, not a theorem about ridge [S:49:251].

Where the analogy breaks. "Exchange rate" suggests a fixed conversion between fit and modesty. The charge $\lambda$ carries the units of the squared answer over the squared coefficient, so it moves when you rescale the answer or the columns. Doubling the recorded counts changes the useful charge. That is why the method asks for standardised columns, and why comparing $\lambda$ across two tables compares two different prices.

Original sources. ISLP section 6.2.1 writes the penalised criterion, explains the exchange between fit and coefficient size, and notes that the whole path costs about as much as one least-squares fit [S:49:248] [S:49:252]. It gives the Bayesian reading at the end of section 6.2.2, one section later in the lasso discussion: ridge is the posterior mode under a Gaussian prior on the coefficients, and also the posterior mean [S:49:259]. The Elements of Statistical Learning adds that the lasso, unlike ridge, is not the posterior mean [S:64:91].

An open question. Nothing in the method chooses the charge. Cross-validation supplies an estimate, and the curve is often flat near its minimum, so several charges fit the same table about equally well. The corpus answers with a convention: take the smallest model whose error lies within one standard error of the best [S:49:248]. That rule is a policy, not a result. Evidence that would settle a case: a selection rule whose test error is bounded without knowledge of the true coefficients, validated on tables where the best charge is known by construction.

A frontier result. The bias-variance curve that L1 draws is not the whole story. As capacity increases toward the point where the model can fit its training data exactly, test error rises and peaks at that interpolation threshold; past it, as the model gains enough capacity to interpolate, test error falls again. The double-descent study reports that shape both as a function of model size and as a function of training epochs [S:311:1] [S:311:2]. The falling branch is the part that contradicts the classical U-curve, so the position of the peak relative to the threshold is the content of the result.

The ridge path is a capacity axis in the linear model, but its λ → 0 end reaches interpolation only when the table has more columns than rows. There the least-squares problem has infinitely many zero-residual solutions and the limiting ridge fit is the one of smallest norm; with fewer columns than rows the path ends at the ordinary least-squares fit, whose training error is generally positive, so that end is not the interpolation threshold [S:49:237] [S:49:251]. The frontier result still says the far end of the path deserves a second look rather than automatic distrust.

What to read next. Read ISLP section 6.2.1 for the criterion, the path and the standardisation argument [S:49:248], then section 6.2.3 for how the charge is chosen. Read the Elements of Statistical Learning's section 3.4 for the prior interpretation and the comparison with best-subset selection [S:64:91]. Then read the double-descent paper for what happens past the interpolation point [S:311:1].