Practise: Neural Networks: Simple Parts, Stacked

At $(0,0)$ the score is $0-0-0.5=-0.5$, which is below zero, so the answer is $-1$. At $(1,1)$ it is $2-1-0.5=0.5$, above zero, so $+1$. At $(2,0)$ it is $4-0-0.5=3.5$, so $+1$. At $(0,2)$ it is $0-2-0.5=-2.5$, so $-1$. The answers are $(-1,+1,+1,-1)$. The boundary is the line $2x_1-x_2=0.5$, because that is where the score is zero. The bias $-0.5$ moved the line off the origin; without it the boundary would pass through $(0,0)$ and the first point would sit exactly on the tie. Section 19.1 sends that tie to $-1$, and here the negative score makes the same choice without needing the convention.

Before the update the score of $x$ is $w\cdot x+b$. After the update it is $(w+x)\cdot x+b=w\cdot x+x\cdot x+b$, so the rise is exactly $x\cdot x$, the squared length. For $(3,4)$ that is $9+16=25$. For the rescaled point it is $0.36+0.64=1$. The rescaled point is the original multiplied by $0.2$, and its squared length is smaller by the factor $0.2^2=0.04$, which is $1/25$. So the same table rescaled by $0.2$ produces score rises $25$ times smaller and weight corrections five times smaller, and the two runs can take different numbers of updates. Section 19.1 names this dependence on the scale of the data. A rule whose correction is the input itself changes with the units in which the inputs are measured, which is why rescaling the inputs changes the sequence of weights.

At threshold $0.5$ the scores $0,2,1,3$ become the answers $0,1,1,1$. The first three rows match the labels and the last row does not, so there is one error. At threshold $1.5$ the answers are $0,1,0,1$: the row $(1,0)$ should be $1$ and the row $(1,1)$ should be $0$, so two rows are wrong. At threshold $2.5$ the answers are $0,0,0,1$: the rows $(0,1)$ and $(1,0)$ lose their $1$, and the row $(1,1)$ keeps a $1$ it should not have, so three rows are wrong. The best of the three thresholds is $0.5$, with one error. This is the pattern section 19.2 describes. Moving the bar up repairs one row and damages another, and no single bar repairs both diagonals at once. A threshold of $0.5$ is the closest this unit comes, and it still misses the row where both inputs are on.

The hidden layer holds four units, and each unit needs three weights and one bias, so it holds $4$ numbers. The hidden layer therefore holds $4\cdot 4=16$ numbers. The output layer holds two units, and each unit needs four weights and one bias, so it holds $5$ numbers. The output layer therefore holds $2\cdot 5=10$ numbers. The total is $16+10=26$. The bias counts because it is a number the training loop adjusts, exactly as a weight is. Section 19.3 counts the smaller network of the chapter the same way: two hidden units with two weights and one bias each give $6$, one output unit with two weights and one bias gives $3$, and the total is $9$. The count grows with the product of the layer sizes, not with their sum, which is why a wider hidden layer is the expensive choice.

The derivative formula gives $${\mathrm{sig}}^{\prime }(1.5)=0.8176\cdot (1-0.8176)=0.8176\cdot 0.1824=0.1491$$ to four decimal places. At $z=0$ the sigmoid is $0.5$ and the slope is $0.5\cdot 0.5=0.25$, which is the largest value of the product $s(1-s)$ for $s$ between $0$ and $1$. The slope at $1.5$ is about $60$ per cent of that maximum. The unit is still responding, but less strongly than at zero, because the sigmoid is flattening as its input grows. Push the input to $z=4$ and the slope falls to $0.0177$, about one fourteenth of the maximum. Section 19.4 uses that flattening to explain why a stack of sigmoid layers weakens the signal that travels back through it.

The composition sends $z$ to $$\frac{1}{2}(-2z+3)-1=-z+\frac{3}{2}-1=-z+\frac{1}{2}.$$ The slope is $-1$ and the intercept is $\frac{1}{2}$. Check one value: at $z=0$ the first step gives $3$, and the second sends $3$ to $1.5-1=0.5$, which is $-0+0.5$. At $z=4$ the first step gives $-5$, and the second sends $-5$ to $-2.5-1=-3.5$, which is $-4+0.5$. Two linear steps have collapsed into one. Section 19.4 makes the general point: the composition of linear maps is a linear map, so a stack of them adds no new shape to the boundary. The slope can change sign, and the line can move, but the result is still one line.

The loss derivative with respect to the output is $o-1=0.9526-1=-0.0474$. The sigmoid slope at $z=3$ is $o(1-o)=0.9526\cdot 0.0474=0.0452$. The weighted sum depends on $w_1$ through $x_1=1$. Multiplying the three factors gives $$\frac{\partial \ell }{\partial w_1}=(-0.0474)(0.0452)(1)=-0.00214$$ to five decimal places. The same product at $z=0$ would be $(-0.5)(0.25)(1)=-0.125$, which is about $58$ times larger in size. Two effects shrink the gradient here. The unit is close to the target, so the first factor is small, and the sigmoid is near its flat region, so the second factor is small too. Section 19.5 reads a gradient as the product of the error downstream and the input upstream; this example shows how a saturated unit multiplies that product by a small number before it reaches the weight.

The slope at $w$ is $2(w-3)$. At $w=0$ it is $-6$, so the first step gives $0-0.25\cdot (-6)=1.5$. At $w=1.5$ the slope is $-3$, so the second step gives $1.5+0.75=2.25$. At $w=2.25$ the slope is $-1.5$, so the third step gives $2.25+0.375=2.625$. The loss there is $(2.625-3{)}^2=(-0.375{)}^2=0.140625$. The losses along the way are $9$, $2.25$, $0.5625$ and $0.140625$, so each step divides the loss by $4$. The distance from $3$ is multiplied by $|1-2\eta |=0.5$ at every step, because this loss is a quadratic and the learning rate is below $0.5$. Section 19.6 uses the same loss with $\eta =0.1$, where the factor is $0.8$ and the walk is slower. A rate of $0.5$ would land on $3$ in one step; a rate of $1$ would send $w$ to $5$ and keep it bouncing between $1$ and $5$.