Why deep networks stop learning

Suppose eight layers each multiply the gradient by one quarter. That is the largest slope a sigmoid can produce, since its derivative peaks at 0.25 at zero input and falls to zero at both ends, so most units in a real stack multiply by much less [S:69:225]. Eight such layers leave a factor of 0.25^8, which is 1/65536, so the first layer receives about fifteen millionths of the signal that the last layer receives. That is where the gradient goes: the loss sends it back, every layer multiplies it, and in a deep stack it arrives at the first layer either too small to use or too large to survive.

The same arithmetic runs forward as well as backward [S:74:127]. A gradient is the list of numbers that says how the loss changes when each weight moves, and backpropagation computes it by walking backward from the loss through every layer. Each layer multiplies the incoming number by its own local slope, the derivative of that layer's output with respect to its input. The chain rule then turns the whole network into one long product [S:78:249]. A product of factors below one shrinks fast, while a product of factors above one grows fast. !Read the stack from the loss on the right back to the input on the left. Each box multiplies the incoming gradient by its own slope, so four layers at 0.25 leave 0.0039 of the signal and eight leave about fifteen millionths. The bars are drawn to scale, which is why the last one is a stub. Status: schematic.

A slope shows up in a small example: a layer that doubles its input has slope 2, a layer that keeps a tenth of it has slope 0.1, and a layer that outputs a constant has slope 0. Stack the last two kinds and the product collapses, while a stack of the first kind grows without limit. !A slope of one is the only value that survives depth: ten layers at 0.25 leave 0.00000095 of the gradient, while ten layers at 2 multiply it by 1,024. Below 1 the early layers stop moving; above 1 the update overflows. Status: re-simulated.

A loss is the single number that measures how wrong the network is on one example or one batch, the group of examples processed in one step. Training lowers it by moving each weight against its gradient, so a weight with a tiny gradient barely moves, and a layer full of such weights looks frozen even though the code is running.

The shrinking case has a name. A vanishing gradient is a gradient so small in the early layers that those layers receive almost no training signal [S:78:250]. The growing case is an exploding gradient, and it shows up as a loss that jumps or overflows [S:78:250]. Neither is a coding error, since both follow from multiplying many numbers, and this is the first answer to the chapter's question about what goes wrong in practice.

The usual culprit is the activation function, the fixed nonlinear step applied after each layer's weighted sum. The sigmoid squashes every input into a narrow range, and its slope is largest near zero and tiny at both ends [S:69:225]. A unit stuck at an end contributes a factor near zero, and the product loses another digit. A rectified linear unit passes positive inputs unchanged and is more stable in deep stacks, which is why it became the default choice in practice [S:69:228].

Think of the loss passing a message back along eight people, each repeating it at a quarter of the volume: by the time it reaches the first layer, fifteen millionths of it is left. The comparison stops at the arithmetic, because layers do not choose to whisper and a product can also grow, and in a deep enough stack both failures can appear in the same run along different directions.

Worked example. Take a four-layer chain and give every layer the same local slope, one quarter. Start with a gradient of 1 at the loss end and multiply as you walk back.

After one layer the signal is 0.25, and after four it is 0.0039, so roughly four parts in a thousand survive and the first layer learns about two hundred and fifty times slower than the last. Replace the slopes with 1 for the units that stay on and the product stays at 1, which is why a rectified linear unit helps; §20.2 supplies the factor of two for the units that switch off. Depth is what turns a small slope into a wall, and at a quarter per layer ten layers take the factor below one millionth.

The other direction is equally real: in one demonstration, a hundred random four-by-four matrices with variance 1 were multiplied together, and the entries grew until the largest needed twenty-six digits to write [S:69:226]. No gradient-descent run started from weights of that size converges, which is the claim the source makes; the failure belongs to descent on the unscaled initialisation, not to every optimizer [S:69:226].

The product picture has limits. A real network has many directions, and the factor can shrink along one and grow along another [S:78:251]. A small gradient is evidence of a problem, not proof of one, because a weight that is already correct needs no update. The cure also costs something, since a rectified linear unit gives up the bounded output that made the sigmoid easy to reason about and can switch a unit off for good [S:69:228].

The formal statement. Write the network as a chain of maps, $x_i=f_i(x_{i-1})$ for layers $i=1$ to $L$, and let $\ell$ be the loss. The gradient at layer $i$ is

Read it as: take the gradient at the loss end and carry it back by multiplying one transposed Jacobian per layer. The chain-rule factor is $\partial x_L/\partial x_i=J_L\cdots J_{i+1}$, with $J_i$ the matrix of partial derivatives of layer $i$ with respect to its input. The hypotheses are: the network is a pure chain, $x_i=f_i(x_{i-1})$, with no path that skips a layer; each $f_i$ is differentiable where the product is evaluated, which means everywhere except the measure-zero kinks of a rectifier, where the framework fixes a value; and the loss is differentiable with respect to $x_L$. That the factors are matrices and that the loss is a scalar are conventions of the notation, not conditions on the network [S:78:249].

The derivation. Apply the chain rule to the composition once, then apply it again to the inner composition. Induction on the number of layers gives the product. The step I omit is differentiability at the kink of a rectified linear unit. That kink has measure zero, and every framework fixes a value there, so the product is correct almost everywhere.

The edge case. The norm of a product is not the product of the norms. You have $\Vert AB\Vert \le \Vert A\Vert \Vert B\Vert$, and the gap can be large. A layer whose singular values are 0.1 and 10 has norm 10, and it shrinks one direction while it stretches another. L1's one number per layer summarises the average squared singular value, not the worst direction. The same pass can therefore vanish along one direction and explode along another. L1 flags that fact at its end, and here is the reason.

A second edge case is the diagnosis. A zero gradient defines a stationary point, so a small gradient is not by itself a fault. The usable test compares the gradient norm in the first layer with the norm in the last, and a ratio that falls with depth is the signature.

What L1 hid, and why that was safe there. L1 replaced each Jacobian by a scalar slope. That replacement assumes the weight matrix preserves length and the activation is the only source of shrinkage. The assumption is safe for the saturating sigmoid L1 discusses. It is not safe in general: at criticality a saturating activation still propagates a usable gradient, because the initialisation keeps the product at one [S:78:252]. The real condition is on the singular values of each Jacobian, and the scalar is their average.

Where the analogy breaks. The whispering line passes one number per person. A layer passes a vector, and a matrix can whisper along one direction while it shouts along another.

Original sources. The chain-rule statement and the product of Jacobians appear in Roberts, Yaida and Hanin, section 9.4, where the factor is written as a product of weight matrices and activation derivatives [S:78:249]. Dive into Deep Learning gives the same product and the accompanying warning about magnitudes [S:69:225]. The problem was first noticed in recurrent networks, before feed-forward networks were deep enough for it to matter [S:78:249]. Prince records the historical reading: progress in the 1990s stalled partly because of sigmoid activations [S:74:127].

An open question. Nobody has settled whether a network must sit at criticality, the point where the typical Jacobian neither shrinks nor grows, or whether criticality is one sufficient condition among several. The theory supplies a clean criterion at that point [S:78:251]. It does not supply a measurement on trained networks. The evidence that would settle the question is a study that estimates the singular value spectrum of the input-output Jacobian in networks that train and in networks that do not, at matched architecture and data. If the spectra separate, criticality is predictive. If they overlap, it is a useful starting rule.

A frontier result. The mean-field treatment keeps the product at one through a recursion for the statistics of the pre-activation. A carefully designed initialisation derived from that analysis trains ten-thousand-layer networks without architectural tricks [S:69:228]. It also explains why a saturating activation is not fatal, which the scalar picture cannot [S:78:252].

What to read next. Read section 9.4 of [S:78:249] for the derivation and the criticality argument. Read section 5.4 of [S:69:225] for the same argument in textbook register. Read the section on clipping in [S:66:374] for the remedy that treats the exploding half and leaves the vanishing half alone [S:66:375]. Then read chapter 7 of [S:74:127] for the history. In that order the readings answer four questions: what the product is, why it is stable at one, what to do when it is too large, and why the problem shaped the field.