Practise: Attention, Images and Sequences

The number of stopping positions is $12-5+1=8$. At the first stop the window is 2, 1, 0, 0, 3. Multiply entry by entry: $1\times 2=2$, $0\times 1=0$, $0\times 0=0$, $0\times 0=0$, and $(-1)\times 3=-3$. The sum is $2-3=-1$. The remaining stops give -2, -3, 0, 3, 2, 1 and -2, so the full output is -1, -2, -3, 0, 3, 2, 1, -2. The layer learns five kernel weights and one bias, which is six parameters, and every stop uses the same five weights. A dense layer from the same twelve inputs to the same eight outputs would learn $8\times 12+8=104$ parameters. The shared kernel pays 6 instead of 104 for the same output shape, and it gives up the freedom to treat the two ends differently.

The convolutional layer holds $4\times 5=20$ kernel weights, because each of the four maps owns one five-tap kernel. It adds one bias per map, so the total is $20+4=24$ parameters. The dense layer produces $4\times 16=64$ outputs. Each output sees all 20 inputs, so the weights are $64\times 20=1280$, and one bias per output adds 64 more. The total is $1280+64=1344$. The ratio is $1344/24=56$. The convolutional layer reaches the same output shape with 56 times fewer parameters, because it applies the same five-tap detector at every position. The dense layer can learn a different detector at each position, and that freedom costs the difference.

The first word contributes 1 at step 0. After one update its contribution is $0.8$, after two it is $0.8^2=0.64$, and after six it is $0.8^6=0.262144$. The path is 1, 0.8, 0.64, 0.512, 0.4096, 0.32768 and 0.262144. Rounded to four decimal places, 0.2621 of the first word remains, so 73.8 percent of it has been lost. The factor is below one, so every step removes a fixed fraction and the early word fades. This is the vanishing-gradient case of section 21.2 seen in the forward direction: the same product of factors that shrinks the state also shrinks the derivative that carries credit back to the first word.

The contribution is multiplied by 1.2 at each of twenty steps, so the factor is $1.2^{20}$. Build it in pairs: $1.2^2=1.44$, $1.2^4=1.44^2=2.0736$, and $1.2^5=2.0736\times 1.2=2.48832$. Then $1.2^{10}=2.48832^2=6.191736$, and $1.2^{20}=6.191736^2=38.3376$. Rounded to two decimal places, the first word's contribution has grown by a factor of 38.34. A factor above one makes the state explode, and the same product makes the backward signal explode. The two exercises E1.3 and E1.4 differ only in which side of one the factor sits, and that side decides which failure the recurrence produces.

Routing A gives $0.5\times 2+0.25\times 4+0.25\times 8=1+1+2=4$. Routing B gives $0.1\times 2+0.3\times 4+0.6\times 8=0.2+1.2+4.8=6.2$. Routing B produces the larger output. Both weight lists are non-negative and sum to one, so each output is a convex combination of 2, 4 and 8 and must lie between the smallest and largest value. B sits closer to 8 because it spends 0.6 of its budget on the third word, while A spends half its budget on the smallest value. The values did not move; only the routing changed.

The exponentials are $e^3=20.0855$, $e^1=2.7183$ and $e^0=1$. Their sum is $20.0855+2.7183+1=23.8038$. Divide each by the total: the weights are $20.0855/23.8038=0.8438$, $2.7183/23.8038=0.1142$, and $1/23.8038=0.0420$. The largest weight is 0.844 and the smallest is 0.042, to three decimal places. The three weights sum to one. A lead of 3 in the raw score becomes a ratio of $e^3=20.0855$ between the first weight and the last, which is why the softmax concentrates attention on the top score. Section 21.4 introduces the scale by the square root of the key length to keep such gaps from becoming too large.

The mean is $(1+2+4+5)/4=12/4=3$. The deviations from the mean are $1-3=-2$, $2-3=-1$, $4-3=1$ and $5-3=2$. Their squares are 4, 1, 1 and 4, whose average is $(4+1+1+4)/4=10/4=2.5$. The standard deviation is $\sqrt{2.5}=1.5811$. The standardised first entry is $(1-3)/1.5811=-1.2649$, which rounds to -1.265. The four standardised entries are -1.265, -0.632, 0.632 and 1.265. Their mean is zero and their standard deviation is one, so the operation removes the old location and scale. The block applies this step per position, not across positions, which is why it can run before or after the mix across positions.

Set the two costs equal: $$n^{2}d=kn{d}^2.$$ Both sides contain a factor of $nd$, so divide it out and leave $n=kd$. With $k=5$ and $d=32$, the crossover is at $n=5\times 32=160$. At that length attention costs $160^2\times 32=25,600\times 32=819,200$ operations, and the convolution costs $5\times 160\times 32^2=800\times 1024=819,200$. The two counts agree. Below the crossover the convolution costs more: at 100 items attention costs 320,000 and the convolution 512,000. Above it attention costs more: at 300 items attention costs 2,880,000 and the convolution 1,536,000. The crossover moves with both the kernel width and the vector width, so the comparison depends on the layer's design as well as on the sequence.