Section 21.5 says attention sees a collection of vectors and the block treats every position alike, so a position signal is added before the first block. It does not say what that signal should be. The field has several answers: fixed sinusoidal patterns, learned vectors with one entry per position, relative offsets computed from the distance between two positions, and rotations of the query and key vectors by an angle that depends on position. No scheme wins on every task, and the choice changes what happens when a model meets a sequence longer than the one it trained on.
Take a position: what should a position signal encode, and what experiment would show whether your choice is right?
In your answer: name at least two candidate schemes and the property each assumes; specify the control and the quantity you would measure; say what pattern of results would count for and against each scheme; state one way the experiment could mislead you; and say what the result would imply for a model trained on short sequences and tested on long ones.
Rubric. Tick each criterion your answer meets, then save the self-assessment. The platform does not grade it.
Worked solution
There is no single correct answer, so this is a model of the shape a complete answer takes, not a key. A level-3 answer fixes the data, the architecture, the optimizer and the number of steps before it compares anything. It names two schemes. The first is an absolute signal added to the input: either a fixed sinusoidal pattern, which has no learned parameters and gives every position a distinct vector, or a learned vector per position, which can adapt to the data but adds a row for every position seen in training. The second is a relative signal inside the comparison: the score between two positions depends on their offset, either through a learned table of offsets or through a rotation of the query and key by an angle proportional to position.
The answer then chooses a measurement that separates them. Train every model at a fixed length, test at that length and at longer ones, and plot accuracy against the distance between the two positions that the task requires the model to relate. If relative offsets are the mechanism, accuracy should depend on distance rather than on absolute index, and the model should degrade gradually beyond the training length. If learned absolute vectors are the mechanism, accuracy should fall sharply once the test sequence uses positions that had no trained vector, while a fixed sinusoidal pattern should extrapolate differently because its vectors are defined at every index. A second measurement is the attention weight as a function of distance: a relative scheme tends to produce a pattern that depends on the offset, while an absolute scheme can produce a different pattern at each index.
The answer states what would refute it: if relative and absolute schemes with matched parameter counts produce the same accuracy at every tested length, then the choice of order representation is not what limits length generalization, and some other factor such as the training distribution or the task's locality is doing the work. It names a confound: learned position vectors add parameters that grow with the maximum training length, so a comparison that does not match the parameter count can attribute their effect to order rather than to capacity. A second confound is the test itself: a longer sequence changes both the positions and the number of items competing in each softmax, so accuracy can fall for a reason unrelated to the position scheme. Permuting positions at test time while keeping the content fixed is one control that separates the two.
Finally, the answer connects the choice to the chapter. Section 21.5 says the block treats all positions alike, so the position signal is the only source of order; section 21.6 says attention connects any pair in one step, so the signal decides whether that step can use distance. The answer also costs the experiment: each scheme needs its own training run at each length, and the comparison needs enough runs to separate the scheme from the seed-to-seed variation. A short answer might use three seeds per scheme and two training lengths, which is six runs per scheme before the longer tests, and it should say so rather than report a single pair of runs.