The chain rule is backpropagation

Raise a number to the fourth power after doubling it and adding one: $y=(2x+1{)}^4$. At x $=1$ the inner value is 3, the output is 81, and a small nudge to $x$ moves the output 216 times as fast. Where does 216 come from? It is two local rates multiplied together: the outer rate $4u^3$ at $u=3$, which is 108, and the inner rate 2 from the doubling, so the product is 216. The 27 is $3^3$, part of the outer rate, not a separate factor.

Those two factors are the answer to the question this section asks: how does a change in one input reach the output? A change in the input reaches the output through every step of the computation, and each step scales it by its own local rate. The chain rule says the total rate is the product of the local rates along the path. Nothing is added; the effects multiply. !The same computation read in both directions. Forward, an input of 1 becomes an intermediate of 3 and an output of 81. Backward, each edge carries its local rate, 2 and 108, and the pass multiplies them into 216. One sweep gives the derivative of the output with respect to every node, which is why a network pays a few passes rather than one pass per parameter. Status: schematic.

Write the path in two stages, with $u=2x+1$ as the intermediate value: !A hundred factors multiplied together. At 0.9 the product falls to 0.000027, at 1.1 it rises to 13,781, and the two curves separate by nearly nine orders of magnitude. Nothing in the rule changed, only which side of one the local rates sit on. Depth turns a small bias in the rate into a vanishing or an exploding gradient. Status: re-simulated from the product rule.

That line says the rate of the whole composition is the rate of the outer step times the rate of the inner step, and it generalises to any number of stages [S:27:154]. The notation $dy/dx$ and $dy/du$ is Leibniz's, a reminder that each derivative began as a ratio of small changes; the pieces are not fractions, but the ratios they came from multiply. The rule is not a trick about fractions. The derivative is a limit of a ratio, and the ratio of small changes composes by multiplication, which is why the limit does too.

A concrete nudge shows the scaling twice. Increase $x$ by 0.001 at $x=1$, and the inner value rises by 0.002. The fourth power turns that into a rise of about 0.216 in the output, because the outer rate there is 108. Divide 0.216 by 0.001 and you recover 216, the product of 108 and 2.

This is the arithmetic behind every trained network. A network is a composition of simple operations, and a computer represents it as a computation graph whose nodes are the operations and whose edges carry values [S:72:21]. To get the derivative of the loss with respect to every parameter — holding the others fixed and asking how the loss moves when that one moves — you walk the graph and multiply the local derivatives along each path [S:27:166].

Doing that walk backwards is backpropagation. The forward pass computes the output and stores the intermediate values. The backward pass starts at the loss and carries one number back through each node, multiplying by that node's local derivative as it goes. The backward order is where the saving comes from. Each node's local rate is computed once and reused by every path that passes through it, so when many parameters feed one loss, one sweep gives every derivative instead of one sweep per parameter [S:27:167].

Nothing in the rule is specific to one input. With several inputs, the derivative with respect to each one is a separate product along its own path, and the backward pass computes all of them together. That is why a network's gradient has one entry per parameter, and why the pass costs a small multiple of the forward pass rather than one pass per parameter.

Worked example. Take the logistic sigmoid, the squash that turns any number into a value between 0 and 1 [S:27:166]. Compose it with a doubling: $z=2x$ and $y=1/(1+e^{-z})$, where $e$ is the fixed number near 2.718 from section 10.3. At $x=0$ the intermediate is 0 and the output is 0.5.

The sigmoid's own derivative is $s(z)(1-s(z))$, so at $z=0$ it is $0.5\times 0.5=0.25$, and the doubling contributes 2. Multiply them and the derivative of the whole is 0.5.

The same bookkeeping explains why deep networks are trainable at all. Each parameter sits somewhere on the graph, and its derivative is a product of the local rates between it and the loss. The backward pass computes all of those products in one sweep, sharing the work when paths merge. A network with a million parameters therefore costs about the same as a few forward passes, not a million of them.

The chain rule is exact for differentiable pieces, and that condition is doing real work. A rectified linear unit is the rule that returns its input when it is positive and zero otherwise, $\max (0,x)$; it is locally Lipschitz and differentiable except at zero, where it has a kink [S:72:30]. At that point the Clarke subdifferential is the whole interval from 0 to 1, and an automatic-differentiation library returns one chosen element of it [S:72:71].

The rule also needs the intermediate values, which is why the forward pass must store them and why memory becomes the limit for very deep graphs.

Limitation. The chain rule turns a derivative into a product, and long products are fragile. A hundred factors each slightly below one shrink toward zero, while factors above one explode, so gradients shrink toward zero or blow up, which is the subject of a later chapter. Writing the product out by hand is also impractical for a large network, since the expression grows faster than the graph [S:27:165].

The formal statement. Suppose $g$ is differentiable at $x$ and $f$ is differentiable at $g(x)$. Then $f\circ g$ is differentiable at $x$, and

$$(f\circ g{)}^{\prime }(x)=f^{\prime }(g(x))g^{\prime }(x).$$

That line says the rate of the composition is the outer rate at the inner value times the inner rate [S:22:85]. For a chain of functions the derivatives compose, so $\partial f=\partial f_K\circ \cdots \circ \partial f_1$, and the chain rule is the product of Jacobian matrices [S:72:73].

The derivation. Differentiability of $f$ at $g(x)$ gives $f(t)-f(g(x))=[f^{\prime }(g(x))+E(t)](t-g(x))$, with $E(t)\to 0$ as $t\to g(x)$ [S:22:86]. Put $t=g(x+h)$, divide by $h$, and use continuity of $g$. The bracket tends to $f^{\prime }(g(x))$ and the second factor tends to $g^{\prime }(x)$. The product of the limits is the derivative. That is the whole proof; the only subtlety is that $g(x+h)$ approaches $g(x)$, which continuity supplies.

The edge case. L1 said that nothing is added and the effects multiply. That holds for a chain, where each value has one parent. It fails at a fan-out, where one variable reaches the output along several paths. For $f(x_1(t),x_2(t))$ the rule reads

$$\frac{df}{dt}=\frac{\partial f}{\partial x_1}\frac{d{x}_1}{dt}+\frac{\partial f}{\partial x_2}\frac{d{x}_2}{dt}.$$

That line says the two path products are added, so a shared input collects one term per route to the output [S:27:154]. A graph with a shared weight is the fan-out case, and the sum at a merge is the addition L1 denied. Every autodiff library implements that sum at the node where the paths meet. L1's sentence is about a single path, so this refines it; a reader who carries the sentence into a network will miss half the gradient.

What L1 hid. L1 said one backward pass costs a small multiple of the forward pass. The formal statement is sharper: a vector-Jacobian product costs about twice the function, and reverse mode must store the intermediate values, so its memory grows with the length of the chain [S:72:204]. That storage is why checkpointing exists [S:72:230]. For a network with a million parameters, that storage, not the arithmetic, sets the memory ceiling of a training step.

Where the analogy breaks. "Each step scales the change by its local rate" assumes that the local rate exists. A rectified linear unit has a kink, and the software picks a value there. A locally Lipschitz function is differentiable almost everywhere [S:72:70], so the exception is a thin set. Training still meets it.

Original sources. The chain rule for one variable is Theorem 2.3.5 in Trench, with the proof that uses the error form [S:22:85]. The rule itself is older than any computer. The historical note in the deep learning book dates it to Leibniz, 1676, and L'Hôpital, 1696, gradient descent to Cauchy, 1847, and the backward application to dynamic programming in the 1960s and 1970s [S:66:200]. Werbos proposed the neural-network use in 1981, and Rumelhart and colleagues made it work in practice in 1986 [S:66:201].

An open question. Reverse mode must store the intermediate values of the forward pass, so its memory grows with the depth of the graph [S:72:204]. Checkpointing trades extra computation for less memory [S:72:230]. One extreme stores nothing and recomputes everything, which is exact and slow; the other stores everything and recomputes nothing. How to choose a schedule between them for a given memory budget and architecture has no settled general answer. Evidence that would settle a case: a schedule with a proved bound on memory and recomputation, measured against the wall-clock time of the exact gradient on the same model.

A frontier result. Reverse mode on top of reverse mode gives Hessian-vector products without forming the Hessian [S:72:233]. That is the mechanism behind second-order methods and curvature diagnostics at network scale, and it applies the same chain rule to the gradient program. The nesting costs extra memory, because the inner pass must be stored to be replayed [S:72:235].

What to read next. Read Trench §2.3 for the one-variable proof [S:22:85]. Read Blondel and Roulet Chapter 8 for reverse mode, its cost and checkpointing [S:72:204]. Read §6.5 of the deep learning book for the historical account and for why the backward order saves work [S:66:199]. Note that the rule is older than the algorithm; the backward order is the engineering.