A leak is decided by which rows a fitted step read, not by which rows its output is applied to.
Question: How can a pipeline that never trains on a held-out row still know that row's label?
Re-simulated Re-simulated: 50 items, 5,000 columns written without the labels, 100 kept, 10 held out, 200 repetitions, data seed 20261108, split seed 60402 (script data/scripts/c04.s02.L1.py; after Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2nd ed., section 7.10.2). Averaged over the 200 repetitions, the screen fitted on all 50 rows keeps columns with 0.4035 |corr| on the held-out rows and the pipeline makes 0.015 mean errors out of 10; fitted on the 40 training rows only, the same pipeline makes 5.03 and its mean |corr| falls to 0.2748, against a mean chance level of 0.2736. A rule that ignores every column makes 5.065. The published 3 per cent differs because 10 held-out items let one item move the count.
Frame 1 of 6: The settingAll 50 rows, 100 columns kept, recorded split 1. The gold chips are the 10 held-out rows. A screen keeps the columns most correlated with the label, and a one-nearest-neighbour rule trained on the 40 navy rows is scored on the gold rows.
The setting. All 50 rows, 100 columns kept, recorded split 1. The gold chips are the 10 held-out rows. A screen keeps the columns most correlated with the label, and a one-nearest-neighbour rule trained on the 40 navy rows is scored on the gold rows.
Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.
Text description of this lab
Fifty row chips: 40 navy training rows and 10 gold held-out rows, with a bracket over the rows the screen reads. Below, one mark per kept column gives its correlation with the held-out label against a chance line; a curve plots mean errors from 10 to 200 columns and a scoreboard reports errors on the ten and the mean over 200 repetitions. All fifty fitted: the cloud sits right of chance, mean 0.015. Forty fitted: back to chance, mean 5.03.