Deeper

Lab · c17.s02.L1

Learning-rate extremes

With the downhill direction already fixed, the step size alone decides whether the run creeps to the bottom, overshoots it while still converging, or grows without bound.

Question: What happens when the step is too small or too large?

Learning-rate extremesA navy bowl draws f(theta) = theta^2/2, starting at theta = 1. Each step adds a gold dot at the next theta and joins it to the previous; a readout gives theta, loss and 1 - eta. At 0.05 the dots creep to the bottom; at 1.5 they zigzag across it with shrinking gaps; at 2.5 they leave the -3 to 3 window, marked by an arrow. A strip below plots theta against step number.Learning-rate extremesupdate theta_next = theta - eta*theta · f(theta) = theta^2/2 · start theta = 1-3-1.501.5301.534.5thetatheta against the step number (same window)30-305101520eta = 0.5 · step 0 of 20 · theta = 1 · f(theta) = 0.51 - eta = 0.5: the sign holds, the distance shrinks by 0.5 each step.
Re-simulated Re-simulated: every run is recomputed from theta_next = theta - eta*theta on f(theta) = theta^2/2 by data/scripts/c17.s02.L1.py, and every number shown is read from data/c17.s02.L1.json. The frames follow one rate at a time from eta = 0.05 to 2.5: the dots creep to the bottom, zigzag across it with shrinking gaps, alternate between two points, or grow out of the -3 to 3 window.
Lab controls
eta = 0.5

Up/Down or Left/Right move one rate; Home goes to 0.05 and End to 2.5

600 ms per step

Left/Right change the delay by 300 ms; Home 600, End 2400

Frame 1 of 12: Step 0: the starteta = 0.5. Start: theta = 1 and f(theta) = 0.5. The downhill direction is already fixed; only the step size is still free.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A navy bowl draws f(theta) = theta^2/2, starting at theta = 1. Each step adds a gold dot at the next theta and joins it to the previous; a readout gives theta, loss and 1 - eta. At 0.05 the dots creep to the bottom; at 1.5 they zigzag across it with shrinking gaps; at 2.5 they leave the -3 to 3 window, marked by an arrow. A strip below plots theta against step number.

Return to the section that hosts this lab

Caption status: re-simulated. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.