Practise: Training Deep Networks Without Breaking Them

Start at 1. After the first layer the gradient is $1\times 0.5=0.5$. After the second, $0.5\times 0.25=0.125$. After the third, $0.125\times 0.5=0.0625$. After the fourth, $0.0625\times 0.25=0.015625$. After the fifth, $0.015625\times 0.5=0.0078125$. After the sixth, $0.0078125\times 0.25=0.001953125$. The shortcut groups the six slopes into three pairs of $0.5\times 0.25=0.125$, and $0.125^3=0.001953125$. Rounded to five decimal places, the input sees 0.00195. The last layer receives the full 1 and the first receives about two thousandths of it, so the first layer moves about five hundred times more slowly. This is the vanishing case of section 20.1, produced by a product of factors below one rather than by a bug.

Fan-in is 2500. He initialisation sets the weight variance to $2/n_{\text{in}}=2/2500=0.0008$. The standard deviation is $\sqrt{0.0008}=0.02828$, which rounds to 0.0283, so option (c) is correct. Option (b) is the plain rule $1/\sqrt{n_{\text{in}}}=1/50=0.02$, which ignores the factor of two that the rectified linear unit requires. Option (a) is the plain weight variance, not the standard deviation. Option (d) keeps the weights at unit scale and makes the output standard deviation $\sqrt{2500}=50$. The factor of two is not a separate principle: half the units switch off, so the surviving half must carry twice the variance. Section 20.2 states the rule; this item checks that you apply the right constant to the right activation.

The fan-in is 200 and the fan-out is 800, so $n_{\text{in}}+n_{\text{out}}=1000$. The Xavier variance is $2/1000=0.002$. The standard deviation is $\sqrt{0.002}=0.044721$, which rounds to 0.0447. The two directions want different constants: the forward pass would ask for $1/200=0.005$ and the backward pass for $1/800=0.00125$. Xavier gives 0.002, which is neither, because it balances the sum of the two factors rather than either one alone. Exercise E2.2 derives what that balance costs.

The mean is $(2+4+4+6)/4=16/4=4$. The deviations are $2-4=-2$, $4-4=0$, $4-4=0$ and $6-4=2$. Their squares are 4, 0, 0 and 4, and the population variance is $(4+0+0+4)/4=8/4=2$. The standard deviation is $\sqrt{2}=1.414$. The standardised first activation is $(2-4)/\sqrt{2}=-2/1.414=-1.414$. The four standardised values are $-1.414$, 0, 0 and 1.414, whose mean is 0 and whose standard deviation is 1. The layer has removed the old location and scale; the learned scale and shift of section 20.3 would then stretch and move these numbers to whatever range training prefers.

Let $v$ be the value training writes when the unit is kept. The unit is kept with probability 0.6 and dropped with probability 0.4, so its expected output is $0.6v+0.4\times 0=0.6v$. Setting $0.6v=9$ gives $v=9/0.6=15$. At prediction time dropout is switched off and the unit contributes 9, which matches the expectation training maintained. The factor $1/0.6=1.667$ is the inverted-dropout scale. Without it, training would see an average of $0.6\times 9=5.4$ and the next layer would be sized for the wrong signal.

A residual block adds the input to the branch output, so its local slope is $1+0.25=1.25$. Eight blocks multiply the gradient by $1.25^8$. Compute in pairs: $1.25^2=1.5625$, $1.25^4=1.5625^2=2.44140625$, and $1.25^8=2.44140625^2=5.960464$. Rounded to two decimal places, the factor is 5.96. The skip turns a branch that would have shrunk the gradient by $0.25^8=0.000015$ into a stack that grows it by about six. Growth is the price of the skip: the same factor of 1.25 that protects the backward path multiplies the forward signal, which is why section 20.4 pairs the skip with normalisation.

Let $v$ be the limiting velocity. At the limit $v_t=v_{t-1}=v$, so the recursion becomes $v=0.75v+4$. Subtracting $0.75v$ gives $0.25v=4$, and $v=16$. The same answer comes from the geometric series $4(1+0.75+0.75^2+\cdots )=4/(1-0.75)=16$. The first velocities are 4, 7, 9.25 and 10.9375, so the path climbs toward 16 without overshooting. Momentum multiplies the step by $1/(1-\beta )=4$ here. A coefficient of 0.9 would multiply it by 10, and a coefficient of 0.99 by 100.

The window ending at step 6 contains the losses at steps 4, 5 and 6: 1.6, 1.8 and 1.2. Their sum is $1.6+1.8+1.2=4.6$, and the mean is $4.6/3=1.5333$, which rounds to 1.533. The full smoothed curve is 3.0, 2.5, 2.467, 2.0, 1.933 and 1.533. The raw curve rises at step 3, from 2.0 to 2.4, and at step 5, from 1.6 to 1.8, while the smoothed curve falls at every step after the first. The window hides the two rises because each one is smaller than the variation the window averages over. A shorter window would track the raw curve more closely and would show more of its wobble, which is the trade-off in section 20.6: the window has to be longer than the noise it is meant to remove and shorter than the trend you need to see.