Deeper

Lab · c02.s04.L1

Train/test wall

An error rate measured on examples the rule has already stored is not a measurement: a stored example is answered by its own label, so the score falls to zero while the rule's answers do not improve.

Question: Why must the test examples be kept away?

Train/test wallA vertical wall separates 600 stored examples on the left from 150 test examples on the right. A slider copies test examples across the wall, each joining the stored pile as a gold ring. As crossed examples rise from 0 to 150, the reported test error falls from 10.67 to 0.00 per cent, while the error on examples that never crossed stays near one in ten: each copy answers itself. Wrong answers are open discs with a cross.Train/test wallWhy must the test examples be kept away?Stored pile: 600 examplesTest set: 150 examplesthe wallREPORTED TEST ERRORERROR ON UNSEEN EXAMPLESTRAINING ERRORthe score on the 600 storedstored exampletest examplecrossed copywrong answer10.67%share crossed: 0.00%10.67%16 wrong of 150 unseen0.00%
Synthetic Synthetic: 600 stored and 150 test examples are drawn by data/scripts/c02.s04.L1.py from two overlapping two-class clouds with a fixed seed; no measured data is used. The eight frames step 0, 25, 50, 75, 100, 125, 140 and 150 test examples across the wall. A crossed example is an exact copy in the stored pile, so its own copy is its nearest neighbour at distance zero and is answered by its own label: the reported error falls from 10.67 per cent to 0.00 per cent while the error on examples that never crossed stays near one in ten.
Lab controls
0 test examples crossed

Left or Down: -1; Right or Up: +1; Page Down: -15; Page Up: +15; Home: 0; End: 150

900 ms per frame

Arrow keys on the slider; Home and End to the slowest and fastest

Frame 1 of 8: 0 of 150 acrossNo test example has crossed. The reported test error is 10.67 per cent, 16 wrong of all 150 test items. With nothing copied, the error on the examples that never crossed is the same 10.67 per cent: the two readouts agree.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A vertical wall separates 600 stored examples on the left from 150 test examples on the right. A slider copies test examples across the wall, each joining the stored pile as a gold ring. As crossed examples rise from 0 to 150, the reported test error falls from 10.67 to 0.00 per cent, while the error on examples that never crossed stays near one in ten: each copy answers itself. Wrong answers are open discs with a cross.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.