Deeper

Lab · c17.s05.L1

Noisy steps

A step computed from one example is a noisy estimate of the full gradient: it heads downhill on average but never settles at the answer.

Question: What if each step sees only one example?

Noisy stepsA russet line steps down a chart, zigzagging above and below a dashed navy line that slides smoothly to the batch mean, 1.034. A gold band around that answer, ±0.208 at the default step size and one example per step, marks how far the jumps reach; it narrows as each step averages more examples and vanishes at 100. Dotted gold, the average of 200 runs, follows the dashed line. A strip of dots shows the 100 example values.Noisy stepsa sampled step heads downhill on average but never settles-3-2-10123050100150200250stepwreadout100 example values: mean 1.034, sd 0.454mean 1.034true 1.0one runfull batchaverage of 200 runsbandanswer 1.034Step 250 / 250step size 0.10examples / step 1band ±0.208final gap 0.038inside band 93%200-run dev 0.024run sd 0.104
Re-simulated Re-simulated by data/scripts/c17.s05.L1.py: 100 synthetic examples drawn from N(1.0, 0.5^2), 250 seeded steps per (step size, examples per step) pair, 200 independent runs averaged for the mean path. The frames are stored simulation output, not measured data.
Lab controls
0.10

Arrow keys choose a step size

1

Arrow keys choose a batch size

500 ms per frame

Arrow keys, Page Up/Down, Home/End

Frame 1 of 8: Whole run: step 250 of 250After 250 steps the run ends 0.038 from the batch mean. 93% of its second-half steps lie inside the ±0.208 band, and the 200-run average departs from the dashed path by at most 0.024. This is the whole run; press Play or Step forward to watch it build from step 0.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A russet line steps down a chart, zigzagging above and below a dashed navy line that slides smoothly to the batch mean, 1.034. A gold band around that answer, ±0.208 at the default step size and one example per step, marks how far the jumps reach; it narrows as each step averages more examples and vanishes at 100. Dotted gold, the average of 200 runs, follows the dashed line. A strip of dots shows the 100 example values.

Return to the section that hosts this lab

Caption status: re-simulated. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.