Derive: Attention, Images and Sequences

(a) For $L=1$ the stack is one kernel of width $k$, so $R_1=k=1+1\times (k-1)$. Suppose the first $L-1$ layers produce outputs whose receptive field holds $R_{L-1}$ values. Layer $L$ reads $k$ consecutive outputs of that stack. The first of those outputs reaches $R_{L-1}$ positions, and each further output shifts the window by one position, so the union covers $R_{L-1}+(k-1)$ positions. Therefore $$R_L=R_{L-1}+(k-1)=1+L(k-1).$$ For $k=3$, $R_2=1+2\times 2=5$ and $R_3=1+3\times 2=7$, which matches the counts in section 21.1.

(b) For width three, $R_L=1+2L$. The condition $1+2L\ge 1000$ gives $L\ge 499.5$, so 500 layers. For width five, $R_L=1+4L$, and $1+4L\ge 1000$ gives $L\ge 249.75$, so 250 layers. Check the boundary: 499 layers of width three see $1+2\times 499=999$ values, one short, while 500 layers see 1001. For width five, 249 layers see 997 and 250 layers see 1001.

(c) In the stride-one stack the length stays the same at every layer, and one layer moves information by at most $k-1$ positions. Crossing a row of $n$ items therefore takes about $n/(k-1)$ layers, which is linear in $n$: 500 layers for 1000 items at width three. The logarithm appears when each layer also divides the length by $k$, as a stride-$k$ layer does. After $L$ such layers the length is $n/k^L$, so the stack reaches a single position when $k^L\approx n$, which is $L\approx {\log }_{k}n$ layers. Section 21.1's stack keeps its length and has linear path length; section 21.6's logarithmic entry describes a shrinking stack. The two statements are consistent once the stride is named, and the comparison with attention's single step depends on which convolutional stack the reader has in mind.

(a) For a finite score $s_i$, the exponential $e^{s_i}$ is a positive number. The denominator is a sum of positive numbers, so it is positive too. The quotient of two positive numbers is positive, so $w_i>0$ for every item. The weights also sum to one, which forces each one below one, so every weight lies in the open interval $(0,1)$.

(b) For the scores 2, 1, 0 the exponentials are $e^2=7.389$, $e^1=2.718$ and $e^0=1$, with total 11.107. The weights are 0.665, 0.245 and 0.090, and the smallest is 0.090. For the scores 5, 1, 0 the exponentials are $e^5=148.413$, $e^1=2.718$ and $e^0=1$, with total 152.131. The weights are 0.97556, 0.01787 and 0.006573, and the smallest is 0.00657 to five decimal places. Raising the top score from 2 to 5 cuts the smallest weight from 0.090 to 0.00657, and it raises the largest weight from 0.665 to 0.976.

(c) The claim that attention chooses a word suggests a selection, with the unchosen words receiving nothing. Softmax mixes every item with a positive weight, so an unchosen word still contributes a small part of the output. The weight tends to zero as its score falls further behind, and the tendency is continuous: every finite score leaves a positive share. Section 21.3's "decides how much of each one to read" therefore means a graduated reading, not an on-off switch. The exact zero comes from a mask, which sets the score to negative infinity before the exponential, so $e^{-\infty }=0$. The positivity is also what section 21.4 needs for training: a weight of exactly zero would block the gradient to that item, while a small positive weight still carries a small gradient.

(a) A single map has $D$ rows and $d$ columns, so it holds $Dd$ weights. One head holds three maps, which is $3Dd$ weights. Across $h$ heads the count is $h\times 3Dd=3D(hd)$. Since $D=hd$, this becomes $3D^2$. The head count cancels: eight heads of width 64 and one head of width 512 hold the same number of weights.

(b) The concatenated output has $h$ blocks of $d$ entries, which is $hd=D$ entries per position. The output projection maps $D$ entries to $D$ entries, so it holds $D^2$ weights. The attention sublayer therefore holds $$3D^2+D^2=4D^2$$ weights.

(c) For $D=512$, $D^2=262,144$. The three maps hold $3\times 262,144=786,432$ weights, and the output projection adds 262,144, for a total of $4\times 262,144=1,048,576$. A single head of width 512 would hold the same 786,432 weights for its three maps. Splitting into eight heads changes what each head can represent, not how many weights the sublayer has. For comparison, the feedforward layer at four times the width holds two maps of $512\times 2048$, which is $8D^2=2,097,152$ weights, twice the attention sublayer. That arithmetic supports section 21.5's claim that most of a block's parameters sit in the feedforward layers rather than in attention.