The weights scale a signal; the activation slope decides whether a deep stack keeps, shrinks or kills it.
Question: Why does a nonlinearity change everything?
Re-simulated Re-simulated: the four activations and their slopes are evaluated from their closed forms, and the depth curves are those slopes raised to the layer count, with weights fixed at 1. At z = 2.0 the sigmoid slope is 0.105 and ten layers pass 1.6e-10; the sigmoid never exceeds 0.250 and reaches 9.09e-13 after twenty layers. The tangent reaches 1.000 at z = 0, so twenty layers still pass 1.000 there, but at z = 2.0 it falls below one in a million by layer 6. The rectifier passes 1.000 where z > 0 and exactly 0 where z < 0; at z = 0 its 0 is a library convention, not a derivative. Every value comes from data/scripts/c19.s04.L1.py; no data set and no fitted model is involved. The weights also multiply the gradient and are held at 1 here, so a falling curve is not a claim about a trained network.
Frame 1 of 1: Sigmoid · z = 2.0 · L = 10z = 2.0, 10 layers. The slope at this pre-activation is 0.105, so the stack passes 1.6e-10. Its slope never exceeds 0.250. The product falls below one in a million at L = 7. Weights are fixed at 1, so this is the activation factor alone.
Sigmoid · z = 2.0 · L = 10. z = 2.0, 10 layers. The slope at this pre-activation is 0.105, so the stack passes 1.6e-10. Its slope never exceeds 0.250. The product falls below one in a million at L = 7. Weights are fixed at 1, so this is the activation factor alone.
Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.
Text description of this lab
Three panels. A draws four activation curves over z from -4 to 4: identity, sigmoid, tangent and rectifier, each with its own colour and dash. B draws their slopes from 0 to 1. C uses a log scale from 1 to 1e-20 to plot each slope raised to the layer count, 1 to 20. A gold guide marks the chosen z and depth. Identity stays at 1, sigmoid and tangent fall, rectifier sits at 1 or exactly 0.