Deeper

Lab · c20.s03.L1

Normalisation effect

Normalisation removes the size of the activations and keeps their shape.

Question: Why does rescaling the activations keep the next layer's input range fixed?

Normalisation effectTwo horizontal number lines, one above the other. Four filled dots enter on the top line. A spread control pulls them apart, from a tight cluster near 2 to a wide range from -8 to 12, while the bottom line keeps four rings at 3, 3, 7 and 7. The normalised values stay at -1, -1, 1 and 1, so the outgoing range stays 4.00 until normalisation is switched off.Normalisation effectsubtract the mean, divide by the standard deviationfilled = incoming - ring = outgoingincomingoutgoing-8-8005510101212Turn Normalise off to watch the outgoing range follow the incoming one.Spread 0.10 to 10.00 in steps of 0.10; arrows, Page Up/Down, Home/End.mean 2.00range 2.001.00, 1.00, 3.00, 3.00mean 5.00range 4.003.00, 3.00, 7.00, 7.00normalised -1.00, -1.00, 1.00, 1.00
Schematic Schematic: the four starting activations are the section's example (1, 1, 3, 3), and every displayed value follows the normalisation recipe exactly: subtract the mean, divide by the standard deviation, then apply the learned scale 2 and shift 5. The lab draws the recipe, not a measured training run.
Lab controls
1 SD

Left/Down -0.10, Right/Up +0.10; Page Down -1.00, Page Up +1.00; Home 0.10, End 10.00

Frame 1 of 10: The four activationsFour activations, 1.00, 1.00, 3.00 and 3.00, enter the layer. Their mean is 2.00 and their standard deviation is 1.00. Normalisation turns them into -1.00, -1.00, 1.00 and 1.00; the learned scale 2 and shift 5 then give 3.00, 3.00, 7.00 and 7.00.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

Two horizontal number lines, one above the other. Four filled dots enter on the top line. A spread control pulls them apart, from a tight cluster near 2 to a wide range from -8 to 12, while the bottom line keeps four rings at 3, 3, 7 and 7. The normalised values stay at -1, -1, 1 and 1, so the outgoing range stays 4.00 until normalisation is switched off.

Return to the section that hosts this lab

Caption status: schematic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.