Deeper

Lab · c14.s03.L1

Ridge shrinkage

One penalty dial slides the fitted weights toward zero, trading a slightly worse fit on the sixteen afternoons for better predictions on afternoons the model has not seen.

Question: Why does adding a penalty on the size of the coefficients make predictions on new data better, even though it fits the data you already have worse?

Ridge shrinkageTwo panels share one penalty dial. Left is the plane of the two fitted sun-record weights: at no penalty the point sits at 3.2497 and -2.3076, and as the penalty grows it slides along a curve into the origin. Right, average squared miss against penalty on a logarithmic error axis: the dotted training curve rises slowly, while the new-afternoon curve falls from 0.3563, bottoms out at 0.16354 at step 28, then climbs to 3.18486, so the middle setting predicts best.Ridge shrinkageOne penalty dial, two panels · 16 afternoons, 30 sets of 40 future afternoonsLEFT · WEIGHTS PLANE (x: weight 1, y: weight 2)RIGHT · MISS AGAINST PENALTY (log error)-4-4-2-202244best0.10.20.40.81.63.2060penalty step →best setting · step 28training miss (dotted)new-afternoon miss (solid, dot markers)step 0 of 60Opposite weights nearly cancel: the sample looks well fitted.λ = 0 · weights 3.2497 and -2.3076 · coefficient size 3.9857train miss 0.09835 · new-afternoon miss 0.3563
Synthetic Synthetic: sixteen afternoons, two sun records and thirty sets of forty future afternoons generated by data/scripts/c14.s03.L1.py (seed 16); no café was measured. The ten frames step the penalty λ from 0 to 5000 and then return to the best setting, step 28. With no penalty the new-afternoon miss is 0.3563; at step 28 it is 0.16354, 2.179 times smaller, while the training miss rises only 1.58-fold; at full shrinkage the new-afternoon miss is 3.18486 and the training miss is 4.03372.
Lab controls
0 step of 60 (λ 0 to 5000)

Left/Right move -1/+1 step; Page Down/Page Up move +/-6; Home goes to 0, End to 60

900 ms per frame

Arrow keys

Frame 1 of 10: No penalty (step 0)At λ = 0 the pair of near-duplicate sun records gets opposite weights, 3.2497 and -2.3076, whose contributions nearly cancel: the training miss is 0.09835, the smallest on the path, while the new-afternoon miss is 0.3563.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

Two panels share one penalty dial. Left is the plane of the two fitted sun-record weights: at no penalty the point sits at 3.2497 and -2.3076, and as the penalty grows it slides along a curve into the origin. Right, average squared miss against penalty on a logarithmic error axis: the dotted training curve rises slowly, while the new-afternoon curve falls from 0.3563, bottoms out at 0.16354 at step 28, then climbs to 3.18486, so the middle setting predicts best.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.