The gradient at the input is one local slope multiplied once per layer, so depth is the number of factors in that product.
Question: Where does the gradient go?
Synthetic Synthetic: an idealised stack in which every layer has the same local slope, so the gradient after k layers is exactly slope ** k and the gradient at the loss is 1. Nothing is trained or measured; data/scripts/c20.s01.L1.py evaluates the product for every setting. The frames walk one bar at a time from the loss back through the stack, then give the verdict. At the default 8 layers and slope 0.25 the input receives 1.53e-05, 65,536 times smaller than the loss end (4.82 decades). Four layers at the same slope leave 0.00391 (256 times smaller); sixteen layers leave 2.33e-10 (4,294,967,296 times smaller, 9.63 decades). At slope 1.00 the input holds 1.00 at any depth; twenty layers at slope 1.50 reach 3325, 3,325 times larger.
Frame 1 of 10: Loss end: gradient 1.00The walk starts at the loss, where the gradient is 1.00. Nothing has been multiplied yet.
Loss end: gradient 1.00. The walk starts at the loss, where the gradient is 1.00. Nothing has been multiplied yet.
Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.
Text description of this lab
A row of bars runs from the loss end at the right, where the gradient is 1.00, to the input at the left, on a log axis marked at 1e-3 and 1e3. At 8 layers and local slope 0.25 the bars fall in a straight line from 1.00 to 1.53e-05, 65,536 times smaller. More layers lengthen the fall; a slope of 1.00 holds flat at 1.00; 20 layers at 1.50 climb to 3325.