Attention

The restaurant was too crowded, so it was expensive. You know what "it" means, because the pronoun points back to "restaurant" rather than to the table or the menu, and the number of words in between does not matter. A recurrent network must carry that noun through every word of the clause, and the previous section showed how the signal fades along the way.

Attention takes a different route: a position compares itself with every item it is allowed to read and decides how much of each one to take. In an encoder that set is the whole sequence; in a decoder it is the position itself and everything before it, and the mask is what separates the two [S:217:5]. !Three values, 10, 20 and 30, mixed with two different sets of weights. The weights are a budget that sums to one, so moving mass from the third item to the first changes the output from 25 to 15 while the values stay where they are. Reading a weight of 0.6 as sixty percent of the blend is the whole operation. Status: synthetic.

Attention builds each output as a weighted average of a set of items, with weights that are non-negative and sum to one [S:74:222]. The output is a mixture rather than a selection. In the restaurant sentence, the word "it" should pay attention to the word "restaurant", and that link depends on content rather than on distance [S:74:222]. !Every row is one word's reading of the sentence, and every row spends a budget of one. The row for 'it' puts 0.316 on 'cat', more than twice the 0.143 an even split would give, so the pronoun's link to the noun is set by content rather than by distance. Rows differ from one another because each position computes its own weights. Status: synthetic.

Each item offers a value, which is the content it can pass on once it has been chosen. For each position, the output is a sum in which every value is multiplied by its attention weight. Prince's textbook calls this routing: the same values can be mixed in different proportions to produce different outputs [S:74:223].

The operation is a soft lookup: a rigid lookup opens one drawer and ignores the rest, while attention opens every drawer and takes a proportion of each [S:74:223]. The fraction is the weight, and the model learns how to compute it. The comparison breaks at the cost: a real lookup reads one page, while attention reads every page before it answers.

Worked example. Suppose three earlier words offer the values 10, 20 and 30. A particular position gives them weights 0.1, 0.3 and 0.6. Multiply first, then add: 0.1 times 10 is 1, 0.3 times 20 is 6, and 0.6 times 30 is 18. The output is 25.

Change the weights to 0.6, 0.3 and 0.1 and the same three values give 6, 6 and 3, which sum to 15. The values did not move, but the routing did, so read the weight 0.6 as sixty percent of this blend coming from the third word, and read the sum to one as a budget that is fully spent.

Where do the weights come from? They are computed from the items themselves, so the pattern of connections changes from sentence to sentence [S:74:222]. A convolution fixes the link between two positions by distance, and attention makes that link a learned function of content. Distance is not part of the arithmetic at all, which is the property that lets the first word of a paragraph meet the last.

When the positions doing the reading and the items being read come from the same sequence, the operation is self-attention [S:217:2]. The name distinguishes it from the older case in translation, where one sequence asks about a separate source sentence, although the arithmetic is the same in both cases.

Self-attention applies the same operation at every position. Stack the outputs of all positions and the layer becomes a table of interactions, with one row per position and one column per item [S:74:223]. Draw that table as shades of grey and you have the attention heatmap: dark cells mark large weights, and the configuration shows which words a position consults.

A recurrent network compresses the past into one fixed-width state before that past reaches the present. Attention keeps the items themselves, so the number of things a later position can consult grows with the sequence, and it reads a weighted sum of their projected values [S:74:222]. That sum is still a summary, a blend that may match no single item, but it is a summary over n stored items rather than over one compressed vector [S:396:1]. The model gains a direct path from any position to any other.

Every position computes its own weights, so a sentence of n items produces n rows. Each row is normalised separately, meaning its entries are divided by their own total, so every position spends its full budget on its own collection of items [S:74:224]. Nothing forces two positions to read the same items.

The expense becomes visible in the quantity of weights. There is one weight for each ordered pair of items, so a sequence of length n requires about n squared of them [S:74:223]. Double the sentence and the attention matrix increases by a factor of four, which is why long documents become computationally expensive.

Two limits follow from the definition. The table of weights makes long sequences costly, and the operation has no sense of order: attention sees a collection of items and cannot tell "dog bites man" from "man bites dog" without extra information [S:217:6]. A single weighted average also mixes every relation into one output, because one average can express one relation [S:217:5]. The usual repair runs several attention patterns in parallel, and the next section supplies the arithmetic that turns relevance into those weights.

The formal statement. Take $N$ items with value vectors $v_1,\dots ,v_N$, and one set of weights for each output. Attention returns

$$y_i=\sum _{m=1}^N{a}_{mi}{v}_m,a_{mi}\ge 0,\sum _{m=1}^N{a}_{mi}=1.$$

The line says each output is a non-negative blend of every value, and the blend spends a budget of one [S:74:222]. The hypotheses are: a finite set of items, values in the same vector space, one weight per ordered pair, and a normalisation computed separately for each output.

The consequence. Every output of the averaging step lies inside the convex hull of the values. A point of that form is by definition a convex combination, and the convex hull is the set of all such combinations. A single weighted average therefore cannot produce the difference $v_1-v_2$, or a negation $-v_m$, because each of those generally lies outside the region the values span. The limit binds the mixture, not the layer. The heads' outputs are concatenated and passed through a learned output projection $W^O$, and each value is itself a projection ${\Omega }_v{x}_m$ of an input, so the layer's output is an affine image of a convex combination and can leave that region [S:217:5]. The output projection is one of the linear maps that escapes it.

The second property. Let a permutation reorder the items. The sum runs over all items, and the weight for a pair depends on the pair rather than on a position. Reordering the inputs therefore reorders the outputs and changes nothing else [S:74:222]. State it exactly: attention is equivariant to permutation, not invariant. The layer knows which item is which. It does not know where the item sits.

The edge case. The restaurant sentence in the core is causal. Each word reads the words before it. Unmasked self-attention lets an earlier word read a later one, so one arithmetic describes two different graphs. In a decoder the mask sets the score of every later item to negative infinity before the softmax, so those weights come out exactly zero [S:217:5]. The surviving scores are exponentiated and divided by their own total, which means the surviving weights each become larger and still sum to one. The output remains a convex combination of the legal items, at the same scale, and what the mask changes is the set of items and the split of the budget between them. A row with no legal item has no weights at all, because the budget has nothing to divide.

What the core hid. It hid where the weights come from, which is the next section. It also hid that one set of weights produces one average. A single head mixes every relation in the sentence into one output, so a relation between two items can be diluted by a relation between two others [S:217:5]. That is the argument for several heads, and the core named it without saying why.

Where the analogy breaks. The soft lookup fails twice. It reads every page, which the core said. It also returns a blend that may match no stored page exactly. A lookup that answered with the average of two drawers would be a poor lookup and an accurate description of attention.

Original sources. The transformer paper defines an attention function as a map from a query and a set of key-value pairs to an output, computed as a weighted sum of the values [S:217:3]. Prince derives the same operation from the language problem. Words need connections whose strength depends on the words themselves, and those connections must span long distances [S:74:222]. The self-attention case, where queries, keys and values come from one sequence, is named in the paper's own definition of the mechanism [S:217:2].

An open question. Does a large weight mean the item caused the output? The paper offers attention distributions as evidence that heads learn different tasks and appear to follow syntax and semantics [S:217:7]. A weight is a coefficient in one average, and a coefficient can be large for an item the model would have used anyway. The field has not settled how often the two coincide. Evidence that would settle a given claim: intervene on the top-weighted item by deleting or swapping it, and compare the change in the prediction with the change the weight predicts. A faithful weight should predict the size of the effect.

A frontier result. Multi-head attention runs several attention functions in parallel on separate learned projections and concatenates the results [S:217:5]. The paper's reason is precise: "With a single attention head, averaging inhibits this" [S:217:5]. One average can express one relation, and a sentence carries several. The same operation then crosses domains. The vision transformer applies attention to patches of an image and reports results against convolutional networks [S:229:1].

What to read next. Read §3.1 and §3.2 of the transformer paper for the definition and the three uses of attention [S:217:3]. Read Prince §12.1 and §12.2 for the routing picture and the softmax derivation [S:74:222]. Then read the attention section of the vision transformer for the same operation applied to patches [S:229:1].