Backprop by hand

The network from §19.3 contains nine parameters, and after one example it produces one loss. Which of the nine is to blame, and by how much? The answer is a number for each parameter, called its gradient, and this section computes all nine with pencil and paper.

The network takes two inputs, x1 = 1 and x2 = 2. The first hidden unit weights them by 1 and 1 with a bias of -3, so its weighted sum is 0 and its sigmoid output is 0.5. The second hidden unit weights them by 1 and -1 with a bias of 1, so its sum is also 0 and its output is also 0.5. !The chapter's network with all nine gradients in place. The forward pass gives hidden answers of 0.5 and 0.5, a prediction of 0.5 against a target of 1 and a loss of 0.125. The backward pass multiplies the loss derivative -0.5 by the sigmoid slope 0.25 and then by the inputs, so the output weights receive -0.0625 each and the first hidden weights receive -0.03125 and -0.0625. Each number is a downstream error times an upstream value, and the magnitudes fall by about a factor of four at every layer. Status: re-simulated.

The output unit weights the two hidden answers by 1 and -1 with a bias of 0, so its sum is 0, its prediction is 0.5, and the target is 1. With a squared error of one half the difference squared, the loss is 0.125. The backward pass then carries a share of that loss back to each of the nine.

$$\frac{\partial \ell }{\partial W}=\frac{\partial \ell }{\partial o}\cdot \frac{\partial o}{\partial z}\cdot \frac{\partial z}{\partial W}$$

That line is the chain rule applied to one weight: the effect of a weight on the loss is the product of three smaller effects, taken in the order in which the information flowed. A forward pass therefore does double duty. It produces the prediction, and it stores every intermediate value that the backward pass will need [S:69:223]. Nothing whatever is recomputed on the way back.

Backpropagation evaluates those products in the reverse direction, starting from the loss and finishing at the first layer [S:69:221]. The reverse ordering matters, because each step requires a quantity that the step after it has already produced. Run the sweep forwards and every weight would need its own separate traversal of the network, which is precisely the computational cost that the reverse sweep avoids [S:66:184].

Worked example. Start at the loss and move back one step at a time. The derivative of the loss with respect to the prediction is 0.5 - 1 = -0.5, since the prediction was too small by half. The slope of the sigmoid at 0 is 0.25, so the derivative with respect to the output sum is -0.125. Both output weights receive the hidden answer 0.5 multiplied by that number, which gives -0.0625 each, and the output bias receives -0.125.

Continue into the hidden layer. The first hidden unit receives the output weight 1 times -0.125, and the second receives -1 times -0.125, so the derivatives with respect to the two hidden answers are -0.125 and 0.125. Multiplying each by the sigmoid slope of 0.25 gives -0.03125 and 0.03125 for the hidden sums. The first hidden weights then receive that number times the inputs 1 and 2, which is -0.03125 and -0.0625, and the second hidden weights receive 0.03125 and 0.0625. The two hidden biases receive the sums themselves, -0.03125 and 0.03125, and all nine numbers are known.

Two conventions make the arithmetic safe. The first is to fix a notation and never vary it: every vector is a column, and every gradient has the identical shape as the parameter it belongs to. The second is to check the shape at each step, because a single transposed matrix still produces numbers of the right size and quietly incorrect values. Shape discrepancies are inexpensive to prevent and expensive to diagnose, which is why they deserve a rule rather than a warning.

The units follow the same bookkeeping. A gradient has the same shape as its parameter, and its units are the units of the loss divided by the units of the parameter; when the numbers look strange, check that division.

The honest limit of the procedure is in its name. Backpropagation computes a gradient and does nothing else, and the actual learning is performed by a separate algorithm that consumes that gradient [S:66:182]. It also retains the intermediate quantities from the forward pass until the backward pass has used them, so training requires substantially more memory than prediction, and the requirement grows with depth and with batch size [S:69:222-223]. A gradient is a direction rather than a step, and the magnitude of the step is the subject of the next section.

The formal statement. Write the computation as a directed acyclic graph. Each node $u_i$ holds a value, and each edge from $u_j$ to $u_i$ carries the partial derivative $\partial u_i/\partial u_j$. Fix one output node $\ell$. Define the adjoint of a node as $v_i=\partial \ell /\partial u_i$. The reverse pass computes $$v_j=\sum _{i:j\to i}{v}_i\frac{\partial u_i}{\partial u_j}$$ starting from $v_{\ell }=1$ and visiting nodes in reverse topological order. One pass fills in every adjoint. The cost is proportional to the number of edges in the graph [S:66:187], and the memory is the set of stored intermediate values, which grows with depth and batch [S:72:207].

The derivation. Induct on the reverse order. The adjoint of the output is one by definition. For any other node, every node that consumes its value has already been processed, because the graph is acyclic and we move backwards. The chain rule over all paths from that node to the output is exactly the sum above, and each term in the sum is already known. The induction closes. The same argument shows why a forward sweep fails: each parameter would need its own traversal, and the whole point of the reverse order is to share the work.

The edge case. A parameter used in two places needs both contributions added, not one overwriting the other. Take a weight that feeds two hidden units. Its adjoint is the sum of the two path derivatives, and an implementation that assigns instead of accumulates drops one of them without any error message. Weight sharing in a recurrent or convolutional model makes this the normal case rather than the exception. The Core's nine-parameter example has one path per parameter, which is why the arithmetic there never shows the sum.

What the Core hid. The Core says the backward pass costs about as much as the forward pass. That is a statement about time, and the price is memory. Every intermediate value from the forward pass is held until the backward pass consumes it [S:69:222-223]. A deeper or wider batch therefore raises the memory of training while leaving the cost of prediction untouched.

Where the analogy breaks. The Core asks which of the nine parameters is to blame. The gradient answers a local question: how the loss changes when one parameter moves and the others stay fixed. It is not a causal claim about the network, and a parameter with a large gradient can be irrelevant once the others adjust. Blame, in the sense a reader wants, is a global property of the trained function, and the gradient does not measure it.

Original sources. The modern account is Rumelhart, Hinton and Williams, named by Bishop and Bishop as the source of error backpropagation [S:67:39]. Goodfellow and coauthors give the computational-graph formulation and the linear-cost argument [S:66:183] [S:66:187]. Blondel and Roulet give the general treatment as reverse-mode automatic differentiation on two linear maps, the Jacobian-vector product and its adjoint [S:72:198].

An open question. Memory. The gradient costs about one extra forward pass in time and keeps every intermediate value, so training needs memory proportional to the stored activations [S:72:207]. Checkpointing and reversible layers trade compute for memory [S:72:198], and no result in this corpus fixes the best schedule for a given budget. The trade is not free in either direction: checkpointing repeats part of the forward pass, and a reversible layer constrains the architecture it can be built into. Evidence that would settle it: a schedule with a proved memory bound and a matching lower bound for a stated class of computation graphs.

A frontier result. Reverse mode over reverse mode gives second derivatives, and forward mode over reverse mode reaches the same quantities while storing less [S:72:235]. The construction matters because the memory bill of the first derivative sets the price of the second. A curvature-aware optimizer pays that bill at every step, so the cheaper second derivative changes which methods are affordable rather than which are correct.

What to read next. Read Goodfellow 6.5 for the graph language and for the sentence that makes the cost linear in the edge count [S:66:183] [S:66:187]. Read Blondel and Roulet chapter 8 for the complexity table and for the checkpointing discussion [S:72:198] [S:72:207]. Read the section of Dive into Deep Learning that writes the same recursion out for a one-hidden-layer network, and check each line against the nine numbers in the Core [S:69:221].