Activation functions

Chain two affine steps — straight-line rules that may also carry an offset — and watch what happens to the pair. The first doubles its input and adds one, so it sends z to 2z + 1. The second triples its input and adds two, so it sends u to 3u + 2. Feed the first into the second and the result is 6z + 5, which is another single affine step, and a third step would collapse in the same way. Depth without a bend buys nothing at all.

The bend is the job of the activation function, a fixed nonlinear rule applied to every weighted sum before it moves on. Three functions carry most of the history and most of the intuition. !Two linear steps, composed and then collapsed. The first sends z to 2z + 1 and the second sends that result to 3u + 2, so the pair is the single step 6z + 5, and a third linear step would fold into the same form. Depth without a nonlinearity therefore adds parameters without adding a shape, which is why every layer in the rest of the chapter carries a bend. Status: schematic.

At z = 1 the logistic sigmoid returns 0.731, the hyperbolic tangent 0.762, and the rectified linear unit 1. The first two flatten as z grows and squeeze the line into (0, 1) and (-1, 1); the third holds a slope of 1 above zero and 0 below it. The three rules are sig(z) = 1/(1+e^{-z}), tanh(z), and max(0, z) [S:69:212] [S:69:212-213] [S:69:211]. That fixed rule is what stops the collapse in the opening paragraph, and it is the reason the rest of this section concerns one function per unit. !The three activation functions and their slopes over the same range. The sigmoid and the tangent flatten at both ends, so their slopes fall toward zero, and the largest slopes are 0.25 at zero for the sigmoid and 1 for the tangent. The rectifier holds a slope of 1 for every positive input and 0 for every negative one; the dashed vertical marks the kink, where the value 0 is a software convention rather than a derivative. Status: re-simulated.

$$\mathrm{sig}(z)=\frac{1}{1+e^{-z}},{\mathrm{sig}}^{\prime }(z)=\mathrm{sig}(z)(1-\mathrm{sig}(z))$$

That line defines the sigmoid and gives its derivative in terms of its own output, so a unit that has already computed its output gets the derivative for free.

The three functions agree on the property that matters here, since each is nonlinear and each has a usable slope somewhere. They differ in how that slope behaves as the input grows. The sigmoid and the tangent flatten at both ends of their range, so their slopes fall toward zero, while the rectifier keeps a slope of 1 for every positive input and a slope of 0 for every negative one.

The reason a nonlinearity changes everything is that compositions of nonlinear steps are not themselves one step. A network with a bend between its layers can carve a boundary that is piecewise linear, as the previous section showed with two hidden units and three thresholds. The pieces meet at corners where the hidden boundaries cross, and no such boundary is curved.

A network without one cannot, whatever its depth, because the composition of affine maps is an affine map, and the whole model reduces to one affine map followed by the output activation — a linear model plus an intercept [S:69:210] [S:63:458]. A stack with no nonlinearity has exactly the expressive power of one unit, bias included, however deep it is. The nonlinearity is what makes depth mean anything.

Worked example. Evaluate the three functions and their derivatives at z = -1, z = 0 and z = 1. The sigmoid gives 0.2689, 0.5 and 0.7311, and its derivative gives 0.1966, 0.25 and 0.1966, so the derivative peaks at the middle point. The tangent gives -0.7616, 0 and 0.7616, and its derivative gives 0.4200, 1 and 0.4200. The rectifier gives 0, 0 and 1, and the derivative is 0 below the kink and 1 above it.

The three functions tell the same story in different measures. The largest slope of the sigmoid is 0.25, at zero, and no input produces a steeper one [S:69:213]. The largest slope of the tangent is 1, also at zero [S:69:214].

That last sentence is the cost, and it has a name. In the saturated regions of a sigmoid or a tangent the derivative sits close to zero, so a gradient arriving from later layers is multiplied by a small number and arrives weakened at the earlier layers [S:63:459]. Stack ten sigmoid layers and the factor is at most 0.25 raised to the tenth power, which is about one millionth, and the early layers receive a signal too faint to move them. This is the vanishing gradient, and it is one reason the rectifier displaced the smooth functions; the Research section says how much of the change it explains and what it leaves open.

The rectifier has its own awkwardness, and the honest version of it is a convention rather than a theorem. Its derivative is 1 above zero and 0 below zero [S:69:210-211], and at exactly zero it has no derivative in the usual sense. Libraries choose the value 0 there and proceed, which is a decision about software rather than a fact about calculus [S:69:211-212]. A unit whose weighted sum is negative on every example in a batch therefore receives no gradient from that batch, and it may stay silent for a long time.

The formal statement. Write one layer as $z\mapsto Az+c$ with $A$ a matrix and $c$ a vector. Two such layers compose to $A_2{A}_{1}z+A_2{c}_1+c_2$, which is again one affine map. A stack of any depth without a nonlinearity between the layers is therefore a single affine map, and its matrix has rank at most the smallest rank among the layers. Add a nonlinearity $\varphi$ applied entry by entry and the composition stops collapsing.

The three standard choices come with their derivatives. The sigmoid is $\sigma (z)=1/(1+e^{-z})$ and satisfies ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$ [S:69:212]. The hyperbolic tangent satisfies ${\tanh }^{\prime }(z)=1-{\tanh }^2(z)$ [S:69:214]. The rectifier satisfies ${\mathrm{ReLU}}^{\prime }(z)=1$ for positive $z$ and $0$ for negative $z$ [S:69:211-212].

The derivation. The vanishing-gradient bound follows from the chain rule and one inequality. Each sigmoid factor is at most one quarter, so a product of $L$ such factors is at most $4^{-L}$. With ten layers the factor is about one millionth, which is the Core's number. The bound is a product of per-unit slopes along one path, and the full derivative sums over all paths through the layer, so the bound applies to each path separately rather than to the total.

The edge case. Read the bound as a property of depth and you will draw the wrong conclusion: at $z=0$ the tangent slope is exactly one, so ten tangent layers can pass a signal back undamped [S:69:214]. Decay comes from saturation, and saturation comes from the scale of the inputs and the initial weights, not from the count of layers. For the tangent, whose slope at the origin is 1, small weights keep ten layers near the identity and the product stays at 1. For the sigmoid the maximum slope is 0.25, so ten layers lose a factor of $0.25^{10}\approx 9.5\times 10^{-7}$ even when every pre-activation sits at the origin; small weights only ensure the layers sit at that ceiling rather than below it. A sigmoid with gain $a$, $\sigma (az)$, has maximum slope $a/4$, so only a gain above 4 gives a per-layer factor above 1.

What the Core hid. The maximum slope of one quarter belongs to the sigmoid with unit gain. Write $\sigma (az)$ and the maximum becomes $a/4$, so the constant is a choice rather than a fact. The Core also hid the hypothesis behind universal approximation: the activation must not be a polynomial, a condition due to Leshno and coauthors that the rectifier satisfies [S:66:179]. A polynomial activation of degree $k$ gives a network that is a polynomial of degree $k^L$, which is a restricted family, whatever the width.

Where the analogy breaks. The Core says a bend is what makes depth mean anything. Depth is not what makes a network expressive, since one hidden layer already approximates any continuous function on a compact region [S:66:177]. Depth buys compactness: the same function with far fewer units [S:69:211]. The bend is what lets the deeper network spend those units on composition instead of on width.

Original sources. The sigmoid as a smooth threshold, its derivative, and the tangent are in the Dive into Deep Learning chapter [S:69:212] [S:69:213] [S:69:214]. The tangent is a shifted and rescaled sigmoid, which is why the two derivatives have the same shape and why the maximum slopes differ by a factor of four. The rectifier is attributed there to Nair and Hinton [S:69:211]. Bishop and Bishop record the change that made training possible: replacing the step function with a differentiable activation that has a non-zero gradient [S:67:38].

An open question. Why the rectifier displaced the smooth functions is not settled. The vanishing-gradient account explains part of it. The derivative of the rectifier is one or zero, so that account says nothing about why a network with a slope of zero on half its units should generalize as well as it does [S:69:211-212]. Evidence that would settle it: comparisons at matched parameter count, matched data and matched optimization budget, with the activation as the only change, reporting approximation error and optimization error separately.

A frontier result. Parametrised rectifiers keep a small slope on the negative side, so a unit that would go silent still passes a gradient [S:69:212]. The modification answers the failure the Core names, and it shows the shape of the field's answer: keep the slope of one on the positive side and repair the negative side rather than return to a smooth curve.

What to read next. Read Dive into Deep Learning 5.1.2 for the three functions, their derivatives and their plots [S:69:211] [S:69:214]. Read Bishop and Bishop 1.3.2 for why differentiability was the enabling change rather than a detail [S:67:38]. Read the spectral-bias paper for the claim that the shape of the learned function sets the order in which structure appears [S:308:1]. Then take the universal approximation statement and check which of its hypotheses the rectifier satisfies [S:66:177].