Initialisation

Four hundred inputs feed one unit, and every weight starts with standard deviation 1. The weighted sum then has standard deviation 20, because the variances of the 400 independent terms add and the square root of 400 is 20 [S:69:227]. Six layers of that kind multiply the spread by 20 each time, which is 64 million, so the signal reaches the loss at a size that no squashing function can rescue.

Initialisation is the choice of values the weights hold before training begins. The section question asks how to start so that signals survive, and the answer is to make each layer preserve the variance of what passes through it. Standard deviation is the square root of the variance, so it measures the typical distance of a number from its mean, and it is the quantity the arithmetic below keeps fixed. !With weights of standard deviation 1 the output grows as the square root of the fan-in, so a layer with 900 inputs is thirty times too wide. Dividing the weight standard deviation by the square root of the fan-in flattens the line at 1, and the inset shows the factor of two that keeps it there once half the units switch off. Status: re-simulated.

Fan-in is the number of inputs that enter a unit. If the inputs have variance 1 and the weights have some variance, the output variance is the fan-in multiplied by that weight variance, provided the inputs and weights are independent and both have mean zero [S:63:483]. Set the weight variance to one over the fan-in and the output variance stays at 1, whatever the fan-in is, which is the whole trick of the method.

Two corrections make this practical. The backward pass sees the layer from the other side, so it cares about fan-out, the number of units the layer feeds [S:69:228]. Xavier initialisation, usually called Glorot initialisation after Xavier Glorot, sets the weight variance to 2 divided by the sum of fan-in and fan-out, which balances the two directions [S:69:228]. It is a compromise, not a solution: the forward pass and the backward pass want different constants, and this choice splits the difference.

The second correction is the activation. A rectified linear unit outputs zero for every negative input, so about half the units are off and half the second moment disappears. He initialisation compensates by doubling the variance to 2 divided by the fan-in [S:63:483]. The factor of two is the price of the activation, not a separate principle, which is why the right constant depends on which activation you use [S:63:483].

All weights equal is a bad start. Units with equal weights receive equal gradients and stay identical, so the layer behaves like a single unit however wide it is [S:69:226]. Random weights break that tie, and the variance rule then decides how large the random numbers should be.

Worked example. Here is what the rule does for three widths. The middle column is the output standard deviation when the weights keep standard deviation 1, and the last column is the weight standard deviation that brings the output back to 1.

Read the first row. With 100 inputs and weights of standard deviation 1, the output is ten times too wide, and dividing the weight standard deviation by 10 fixes it. The third column is one over the square root of the fan-in, which is why the numbers fall slowly as the layer widens: a layer with nine times the inputs needs weights only three times smaller.

He initialisation can be checked by hand as well. For 400 inputs it uses a weight variance of 0.005, so the pre-activation entering the rectifier has variance 2. The rectifier acts on that one number, not on 400 units: with a symmetric pre-activation it passes half the second moment, so the output has second moment 1 and a nonzero mean, and its variance is smaller than that. The factor of two therefore restores the second moment, and with it the root mean square. That root mean square is what keeps the signal from shrinking layer by layer; restoring the variance itself would need a slightly larger constant, and the Deepen layer gives it.

A unit is off on about half of its inputs, not half the units are off. The same rule explains the opposite failure: weights with standard deviation 0.01 and fan-in 400 give an output standard deviation of 0.2, and six such layers take it to 0.000064, so the signal fades before it reaches the loss.

Nothing in the argument requires a Gaussian distribution, only a mean and a variance that exist [S:69:227]. That is why the same rule works for uniform weights, after a change of scale, and why the derivation can be written with a Gaussian or a uniform draw.

The derivation assumes independent inputs with mean zero, and it ignores the nonlinearity [S:69:228]. It controls the first step, not the whole run: a deep stack can still drift as training proceeds, and later sections add the devices that hold it in place [S:74:127]. The cost is that the constant depends on the activation, so changing the activation invalidates the number you computed.

The formal statement. Let a unit receive $n$ inputs $x_i$, each with mean zero and variance ${\gamma }^2$, and let its weights $w_i$ be independent of the inputs, with mean zero and variance ${\sigma }^2$. The pre-activation is $z={\sum }_{i=1}^n{w}_i{x}_i$. Then the mean is zero and

Read it as: the variance of the sum is the number of terms times the variance of one term. Setting ${\sigma }^2=1/n$ makes the output variance equal the input variance at every width. The backward pass gives the same statement with the fan-out in place of the fan-in, and Xavier splits the difference with a variance of two divided by the sum of the two counts [S:63:483].

The derivation. Each term has mean zero, because the factors are independent and each has mean zero. The terms are independent of one another, so the variance of the sum is the sum of the variances. Each variance is the product of the two variances, and there are $n$ of them. That is the whole argument. For a rectified linear unit you add one hypothesis: the pre-activation is symmetric about zero with variance ${\gamma }^2$, so the output is zero on about half of the inputs, its second moment is ${\gamma }^2/2$, its mean is $\gamma /\sqrt{2\pi }$ and its variance is ${\gamma }^2(1/2-1/(2\pi ))$. Restoring the second moment costs a factor of two, which is He initialisation [S:63:483]. Restoring the variance itself needs ${\sigma }^2=1/(n(1/2-1/(2\pi )))\approx 2.93/n$, nearly one and a half times larger, because the rectified output has a mean to carry as well as a spread.

The edge case. The recursion controls the variance in expectation and says nothing about a single run. Draw one weight matrix and the realised variance of its output differs from the formula. The difference has a spread that grows with the kurtosis of the weight distribution. Stack many layers and those deviations compound, so two runs from the same rule can end in different regimes. The rule fixes the mean of a random quantity, not the quantity.

A second edge case is the bias. The factor of two assumes a symmetric pre-activation, and a nonzero bias breaks the symmetry. The fraction of active units is then not one half, and the constant that restores the variance is not two. The constant is a property of the activation and the data, not a universal number.

What L1 hid, and why that was safe there. L1 said the argument needs only a mean and a variance [S:69:227]. That is true for one layer and false for a stack. A deep network's output distribution is not fixed by its first two moments, and the independence of the inputs is an assumption the network violates once its units become correlated. The one-layer calculation is safe because the reader needs the mechanism. The correction is the subject of the paragraph above.

Where the analogy breaks. L1's "keep every layer in the same range" treats the layers as independent rescaling operations. They are not independent: they share inputs, and their outputs are correlated. A rule that preserves each layer's variance separately can still change the distribution of the whole stack.

Original sources. Glorot and Bengio proposed the fan-in and fan-out compromise, and He and his coauthors proposed the factor of two for rectified units. Murphy states both rules and names the earlier LeCun rule that they generalise [S:63:483]. Dive into Deep Learning gives the same derivation and names the rule after Glorot, its first author [S:69:228].

An open question. The derivation is linear and assumes independent inputs, yet the rules work on nonlinear networks whose units are correlated. Nobody has settled which property of the initial weight matrix predicts trainability: the variance of the output, the full singular value spectrum, or the depth at which the spectrum acquires a heavy tail. The evidence that would settle it is an experiment that fixes the output variance and varies the spectrum, then measures the depth at which training fails. If the spectrum predicts failure and the variance does not, the variance rule is a proxy that happens to correlate with the cause. A null result would also inform: if neither predicts failure, the depth at which training stops is set by the optimisation of the whole stack, not by the start.

A frontier result. Data-driven initialisation replaces the closed-form rule with a measurement. Layer-sequential unit variance initialises each layer with orthonormal matrices and then rescales by the observed activation variance on one batch [S:63:483]. GradInit learns a scaling factor for each weight matrix, and Fixup removes the need for normalisation in residual networks by re-parameterising the initialisation [S:74:128]. A carefully designed initialisation also trains ten-thousand-layer networks without architectural tricks [S:69:228].

What to read next. Read section 13.4.5 of [S:63:483] for the derivation and the variants. Read section 5.4 of [S:69:227] for the textbook version and the pointer to very deep training. Read chapter 7 of [S:74:127] for the notes that place He and Xavier after LeCun. Then read the identity-mapping paper [S:179:1], because Fixup is the residual-network version of the same question.