Residual connections

Ten blocks in a row each multiply the gradient by 0.5, so the product is 0.00098 and the input end receives about one thousandth of the signal. Add each block's input to its output and every factor becomes 1.5; the same ten blocks then multiply the signal by 57.67, although the blocks themselves did not change and only the wiring did.

The wire that adds each block's input to its output is a residual connection, also called a skip connection. That wire is the whole difference between the two columns: add 1 to the 0.5 slope in the table and the factor becomes 1.5 [S:65:239]. A block with such a wire is a residual block, and what it learns is the residual, the correction added to the input you already have. !The skip adds the block's input to its output, so the derivative of a block is 1 plus the derivative of the correction. Expanding ten such factors gives 1,024 terms, and the one that takes every skip equals 1 no matter what the branches learn. Status: schematic.

The forward pass lets the input travel unchanged along the skip, and the backward pass adds the incoming gradient to the block's own gradient, so each skip contributes a term of size 1 [S:65:240]. A product with many such terms cannot collapse to zero, which is why the gradient still reaches the early layers [S:73:87]. Written out, the derivative of a block is one plus the derivative of the correction, and a product of such terms expands into a sum in which the all-skips term is exactly 1. For ten blocks that sum has 1024 terms, and the all-skips term is present whatever the blocks learn, which is a route that plain stacking does not have.

The version that works best uses an identity mapping for the skip, so the block output is the input plus a correction [S:179:1]. At the start of training the correction is small, which makes every block close to doing nothing, and the skip itself is not a learned parameter but an architectural promise. That matters because of the degradation problem: a deeper plain network can end with a higher training error than a shallower one [S:65:239].

A deeper plain network contains a shallower one in principle, because the inserted block can approximate the identity [S:65:238]. Containment is exact when the identity is exactly representable, which a rectified block manages at twice the width; otherwise it is approximate, and poor for a saturating nonlinearity. So what the plain stack fails at is not representation but optimisation: the deeper network does not find that solution, and a skip makes the do-nothing solution the default.

The blocks still interact, because a residual stack is a shared channel that every block reads and writes, and each block only has to add a small correction to what is already there. The comparison with a relay race breaks at the handover, since no block owns the signal and every block sees the same running total rather than a baton passed to one runner. That shared channel is why residual connections appear in most very deep models, including the transformers of the next chapter [S:73:87].

Worked example. Compare two stacks over the same number of blocks: in the plain stack every block has local slope 0.5, and in the residual stack the skip adds 1 to that slope so every factor is 1.5.

At ten blocks the plain stack has lost three digits and the residual stack has gained one. The residual column grows, and that growth is the price: the skip protects the path but does not keep the size at 1. It is the milder failure: over ten blocks the residual stack multiplies the signal by 57.67 while the plain stack multiplies it by 0.00098. One column gains a factor of 58 and the other loses a factor of 1024, and the growth keeps climbing.

A stack of such blocks is a residual network, and it usually pairs the skip with the normalisation of the previous section. Placed on the residual branch, normalisation biases each block towards the identity at the start of training, and it works with the skip rather than repairing the growth [S:65:239].

The connection has a hard precondition: the input and output must have the same shape, or the skip needs its own rescaling [S:65:239]. The cost is depth, not the skip: the forward pass stores every layer's pre-activations for the backward pass. The same source gives gradient checkpointing, which stores them every N layers and recomputes the rest, and micro-batching as the remedies [S:74:128]. A residual block adds no new kind of stored value, since it adds its input to its output and that input is already kept for the branch's weight gradient. And the growth in the table is real, so a residual stack usually needs normalisation beside it [S:65:239].

The formal statement. A residual block is $x_i=x_{i-1}+F_i(x_{i-1})$, with $F_i$ the residual branch [S:65:239]. Its derivative is the identity plus the derivative of the branch [S:65:240]. For a stack of $L$ blocks, the gradient at the input is

Read it as: the input gradient is the output gradient plus one correction term per block, and the first term does not depend on the depth. This is an identity, not an approximation. The hypotheses are that every branch is differentiable and that input and output have the same shape.

The derivation. Apply the chain rule to one block. The gradient at the input of the block is the gradient at its output plus the branch derivative applied to that gradient. Now substitute that same statement for $g_i=\partial \ell /\partial x_i$ repeatedly: the leading term walks forward one block per substitution and emerges as $g_L$, while each intermediate gradient is left behind as the argument of exactly one branch derivative. Nothing is subtracted, so nothing cancels: the identity terms telescope forward and the corrections accumulate. The step I omit is the existence of the branch derivative, which fails at the kinks of a rectified unit and holds almost everywhere [S:179:1].

Compare the plain stack. There the input gradient is a product of derivatives, and a product of small factors is small. The residual form is a sum, so the identity term survives whatever the branches do. Bishop makes the same point from the forward direction: the expanded composition is a sum of sub-networks of different depths [S:67:293].

The edge case. The identity needs equal shapes. When the input and output differ in width, the skip becomes a learned map, and the first term of the sum is that map applied to the output gradient [S:65:239]. The guarantee now depends on the smallest singular value of the learned map. An ill-conditioned projection can shrink the clean path, and the skip is no longer free. A second edge case is the range of the branch. A block that ends in a rectified unit can only add a non-negative correction, so every coordinate of the block's output is at least the corresponding coordinate of its input [S:65:240]. The "do nothing at the start" story needs the branch to be able to output zero, and that is a design constraint.

What L1 hid, and why that was safe there. L1 says a deeper plain network contains a shallower one in principle, because the inserted block can approximate the identity. Containment is a statement about the function class, and its strength depends on how well the inserted layers represent that identity. A residual block represents it exactly when the branch outputs zero [S:67:292]. A plain rectified block represents it exactly too, at twice the width: the map $x\mapsto W_2\mathrm{ReLU}(W_{1}x)$ with $W_1=[I;-I]$ and $W_2=[I,-I]$ is the identity on ${\mathbb{R}}^n$. For a saturating nonlinearity the plain block only approximates the identity, and the approximation is poor wherever the units sit near saturation. So the failure that matters is optimisation, not representation: both function classes contain the shallower model, and only the residual parameterisation makes that solution easy to reach. The worked example used constant slopes, where the containment is exact, so the qualifier never showed.

Where the analogy breaks. The relay-race comparison fails at the handover, as L1 says. A second break is the size of the correction. A baton is either passed or dropped, while the residual sum can be dominated by its corrections even when the identity term is present.

Original sources. The residual block is from He, Zhang, Ren and Sun in 2015, and Scardapane gives the definition together with the vector-Jacobian product that makes the backward path exact [S:65:240]. The identity-mapping analysis is the same authors in 2016. It shows that the forward and backward signals travel directly from one block to any other when both the skip and the after-addition activation are identities [S:179:1].

An open question. Nobody has settled why residual networks generalise better than plain networks of the same depth and parameter count. The gradient-path argument accounts for trainability, not for test error. One candidate is the ensemble reading: the expanded network is a sum of sub-networks of many depths, and the prediction averages them [S:67:293]. What would settle it is an ablation that removes the skips at test time and measures the change in error, together with a measurement of how much each path contributes. If the shallow paths carry most of the prediction, the ensemble story has content. If the deepest path dominates, the story is decoration. The same question applies to depth without skips: a plain network can represent the identity in principle, and it fails in practice, so part of the gap is about optimisation rather than representation.

A frontier result. Identity mappings make the path exact. With pre-activation and an identity skip, a network of a thousand and one layers trains and improves on the original design [S:179:1]. The same block is the unit of the transformer, which stacks attention and a small network inside residual connections [S:73:87].

What to read next. Read chapter 9 of [S:65:239] for the block and the shape condition. Read the identity-mapping paper [S:179:1] for the ablation. Read section 9.5 of [S:67:293] for the expansion into sub-networks. Then read section 4.7 of [S:73:87] for the place of the skip in modern architectures.