Normalisation and regularisation

Four activations leave a layer as 1, 1, 3 and 3. Their mean, the ordinary average, is 2, and their standard deviation is 1. Subtract the mean and divide by the standard deviation, and they become -1, -1, 1 and 1. Multiply by 2 and add 5, and they become 3, 3, 7 and 7. The layer has thrown away its old scale and taken a new one that you chose.

The section asks two questions: why rescale the activations, and when should you drop units. Batch normalisation performs that recipe at every layer, using the statistics of the current batch, the small group of examples processed in one step. It rescales because the next layer's weights were sized for a particular range, and keeping that range fixed lets you use a larger learning rate [S:69:334]. The second question has a short answer too, since units should be dropped when the training error falls while the validation error rises. After standardising, the layer applies a learned scale and a learned shift, so it can undo the change if the data prefer a different range [S:69:334]. !The four activations keep their spacing at every step. Subtracting the mean of 2 and dividing by the standard deviation of 1 centres them on zero; a learned scale of 2 and shift of 5 then set a mean of 5 and a standard deviation of 2. The bracket under each row spans one standard deviation. Status: re-simulated.

Scale matters because a weight update is measured against the size of what passes through it. If every weight in a layer doubles, every pre-activation, the weighted sum before the activation, doubles, and the same learning rate now moves the next layer twice as far. A range that drifts from layer to layer therefore turns one global learning rate into a different effective rate in each layer, and the layer whose range is largest sets the limit for the whole run. !In training half the units are dropped and each survivor carries twice its activation, so a value of 6 contributes 0.5 x 12 = 6 on average. At prediction every unit is present at 6, which is why training and deployment see different widths of the same layer. Status: schematic.

A layer whose outputs drift to a variance of 100 or 0.01 keeps the same shape, and normalisation removes the size while preserving the shape. The small constant added under the square root keeps the division safe when every value in the batch is equal [S:69:334].

Why it helps was first explained as internal covariate shift, the idea that each layer keeps facing a moving input distribution. That explanation has not held up [S:69:341]. What survives is that the scale is under control. The smoother-landscape story is the leading intuition, not a measured effect: the source says batch normalisation "is thought to make the optimization landscape smoother" and asks the reader to separate such intuitions from established fact [S:69:340]. The lesson generalises, since a mechanism can work for reasons other than the ones in its first paper.

The second question is about dropping units. Regularisation is any penalty or constraint that stops a model from fitting noise. Dropout removes each hidden unit at random during training, keeping it with probability 0.5. The paper finds that value close to optimal for a wide range of networks, and for the visible units the optimal retention is usually closer to 1 than to 0.5 [S:172:2].

A kept unit is doubled, so its expected output is unchanged: an activation of 6 becomes 12 half the time and 0 the other half, with mean 6. The network therefore trains on a different thinned architecture at every step, and no unit can rely on a particular neighbour being present [S:172:1].

Drop units when the model overfits, which means its training error sits far below its validation error [S:69:155]. Dropout also slows training, because every step sees a weaker network, and it is switched off at prediction time, so the model you deploy is not the model you trained on any single step [S:172:1].

Worked example. Start with the four activations above and apply the recipe step by step. The last column uses a scale of 2 and a shift of 5.

The mean of the last column is 5 and its standard deviation is 2, which is exactly the scale that was asked for. The recipe learned nothing at this point; it only moved and stretched the numbers. Training then adjusts the scale and the shift, and the layer can recover any range it needs.

The cost is that batch statistics differ between training and prediction, so the layer behaves differently in the two modes [S:69:335]. A batch of one has no spread to divide by [S:69:334], and sequence models prefer layer normalisation, which standardises each example on its own and therefore needs no batch at all [S:69:336]. Dropout carries the same train-and-test split, and it spends training time on a noisier gradient [S:172:1].

The formal statement. For a minibatch $B$ with mean ${\mu }_B$ and variance ${\sigma }_B^2$, batch normalisation computes

with a learned scale $\gamma$, a learned shift $\beta$ and a small positive constant $\epsilon$ [S:69:334]. Read it as: centre the batch, divide by its spread, then apply a learned scale and shift. Dropout draws a mask $m_i$ from a Bernoulli distribution with keep probability $p$ and returns $m_i{x}_i/p$. Read that as: keep a unit with probability $p$ and multiply the survivors by one over $p$, so the expected value is unchanged.

The derivation. The dropout expectation is one line: the expectation of the mask is $p$, so the expectation of the output is the input. The variance is not preserved, and the correction is the point. The output variance is the input squared times one minus $p$ over $p$, which grows as $p$ falls. The original paper uses the other convention, keeping the training activation and multiplying the weights by $p$ at test time. The two conventions agree in expectation [S:172:1].

For normalisation, the mean and variance of the output are the shift and the square of the scale by construction. That is what "the next layer sees a fixed range" means. The operation is not linear in the input, because the batch mean and variance depend on the input. Its derivative is the scale over the batch standard deviation times the identity, with two rank-one corrections, and those corrections are why normalisation changes the gradient and not only the scale.

The edge case. A batch of one. Then the batch mean is the input and the batch standard deviation is zero, so the numerator is zero and the constant decides the answer. The layer returns the shift, and the derivative with respect to the input is not the identity. Dive into Deep Learning states the consequence plainly: with minibatches of size one you cannot learn anything [S:69:334]. Layer normalisation avoids the case by standardising across the features of one example [S:69:336]. A second edge case is the train-and-predict split: the layer uses batch statistics in training and running statistics at prediction, so one set of weights produces two functions [S:69:335].

What L1 hid, and why that was safe there. L1 said normalisation removes the size and preserves the shape. That is true of the values and false of the map: dividing by a batch statistic couples every example in the batch to every other. The coupling is why the layer behaves differently at prediction, and why a batch of one has no answer. In the worked example the batch was the whole table, so the coupling never showed.

Where the analogy breaks. "Throw away the old scale" suggests that information is lost. The learned scale restores any size, so what the layer discards is the batch's own scale, not the relative pattern of its values.

Original sources. Batch normalisation comes from Ioffe and Szegedy in 2015, and dropout from Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov in 2014 [S:172:1]. The textbook here summarises and qualifies the normalisation paper [S:69:334]. The retention probability of one half is the value the dropout paper recommends for a wide range of networks [S:172:2].

An open question. The original explanation of batch normalisation was internal covariate shift: each layer keeps facing a moving input distribution. That explanation has not held up [S:69:341]. The surviving account is that normalisation smooths the optimisation landscape [S:69:340], and that account is an intuition rather than a theorem. What would settle it is an intervention that holds the input distribution of each layer fixed while removing the normalisation, at a matched effective step size. If the benefit survives, covariate shift was never the cause. If it disappears, the smoothing story needs a mechanism. The literature also mixes two claims: that normalisation changes the optimisation problem, and that it changes the statistics of each layer. Separating them needs an experiment that holds one fixed while it varies the other.

A frontier result. Layer normalisation replaces the batch statistic with a statistic of one example, so the layer needs no batch and behaves the same in training and prediction [S:69:336]. That property is what makes it usable in sequence models, and it is why the transformer block of the next chapter normalises each token rather than each batch.

What to read next. Read section 8.5 of [S:69:334] for the definition, the train-and-predict split and the critique. Read the dropout paper [S:172:1] for the original motivation and the test-time scaling rule. Read it beside the normalisation pages, because both make training stochastic and both change behaviour at test time. Read the discussion in [S:69:340] for the honest statement of what is not known. Then read the identity-mapping paper [S:179:1], because normalisation and residual connections are designed together.