The transformer block

A transformer block reads a sequence of eight vectors, each 512 numbers long, and returns eight vectors of the same size [S:217:5]. Nothing is compressed and no position is dropped, so the next block receives exactly the shape the first one returned. That stability is what allows a model to stack many blocks.

The block does two jobs in alternation. Attention lets positions exchange information, and a small network then processes each position on its own [S:217:5]. Both results are added back to the input, and a rescaling step follows each addition [S:74:230]. The repeating unit of a transformer is this pair of operations plus the connections between them. !One block, drawn in the order it runs. Attention mixes the six position vectors, the addition keeps the input, the normalisation rescales each vector, and the feedforward layer rewrites each position on its own before a second addition and normalisation. The shape entering and the shape leaving are the same, which is what lets a model stack sixty of these. Status: schematic.

In symbols the block is short. For a matrix $Z$ whose columns are the position vectors,

Read the first update as two steps. The attention term computes a correction for each position, and the sum adds that correction to the input itself; the addition is a residual connection [S:74:230].

A residual connection lets a block learn a small correction instead of a whole replacement, while the original signal stays available. Chapter 20 showed why that makes deep stacks trainable: the signal has a path to the output that skips every sub-layer. The norm then rescales each position vector on its own. The second line applies the same two steps to the small network, which this section calls the feedforward layer.

Worked example. Take the four numbers 1, 2, 3 and 4. Their mean is 2.5, so subtracting it gives -1.5, -0.5, 0.5 and 1.5. The squares are 2.25, 0.25, 0.25 and 2.25, and their average is 1.25. The square root of 1.25 is about 1.118, and dividing the centred numbers by it gives -1.342, -0.447, 0.447 and 1.342.

That subtraction of the mean and division by the spread is layer normalisation, applied to each position vector separately. For the parameter-free normaliser the output has mean zero and standard deviation one by construction, and the map is unchanged by adding a constant to every entry or by multiplying the vector by a positive constant [S:180:3]. The layer norm inside the transformer then multiplies by a learned gain and adds a learned bias, so the block's output has that bias as its mean and a spread set by the gain [S:180:3]. What is pinned to zero and one is the input to that last map, not the output of the sub-layer.

The attention sublayer is multi-head attention. Each head projects the position vectors into its own smaller space, runs attention there, and the results are concatenated and projected back [S:217:5]. With eight heads and a model width of 512, where the model width is the number of entries in each position vector, each head works in 64 dimensions. That is an eighth of the entries, and the total cost stays close to that of one full-width head.

The second sublayer is a feedforward layer, which applies two linear maps with a nonlinearity, a bend applied to each number, between them. It treats each position separately, so it cannot mix information across positions [S:217:5]. What it can do is transform each vector, and it works in a wider space: the model width of 512 becomes 2048 inside the layer, which is a factor of four [S:217:5].

The division of labour is the point. Attention moves information between positions, and the feedforward layer computes with whatever arrives at one position. A stack alternates the two, so a feature built at one position can be sent to another and then transformed there.

A block that predicts the next item must not read the future. A mask sets the scores of later positions to negative infinity, so a position can read only itself and the positions before it [S:217:5]. The mask uses position to forbid, but neither attention nor the feedforward layer reads position from the vectors. Order is not part of either operation, so the model must be told where each item sits, and a position signal, a vector that encodes the index, is added to the input before the first block [S:217:6]. Without it, shuffling the items would leave the output unchanged.

Stacking is composition. Block one returns a matrix of the same shape, block two reads it, and after sixty blocks every vector has been rewritten sixty times. The parameter count per block is fixed, so depth is one way the model grows, alongside width.

The block does not decide when to stop, and it does not produce words. It rewrites vectors, and a final output layer turns the last vector into scores over the vocabulary [S:217:5]. The same block, without the mask, can read the whole sequence at once.

The cost grows with depth. Every block pays for an attention table and a feedforward pass, and the table grows with the square of the sequence length. The wider inner layer is four times the model width in the original design, so most of the parameters sit in the feedforward layers rather than in attention [S:217:5]. The residual stream, the running sum that every block adds to, is a useful picture of what flows through the block, and it is a picture rather than a proof.

The formal statement. The 2017 block applies two sub-layers, each wrapped in a residual connection and followed by a normalisation [S:217:3]:

$$z_i=\mathrm{norm}\left(x_i+a_i\right),y_i=\mathrm{norm}\left(z_i+m_i\right),$$

where $a_i$ is the attention output at position $i$ and $m_i$ is the feedforward output [S:74:230]. The line says each sub-layer adds its result to its input and the sum is normalised, once for attention and once for the small network. The hypotheses are: attention mixes positions, the feedforward layer does not, the normalisation is applied to each position separately, and every sub-layer returns the width it received.

The derivation. For a position vector with entries $z_1,\dots ,z_D$, let $\mu$ be the mean and $\sigma$ the standard deviation. The block replaces entry $i$ by $(z_i-\mu )/\sigma$. The new mean is zero, because the centred entries sum to zero. The new standard deviation is one, because the sum of squared centred entries is the width times the variance, by definition. That is the whole derivation, and it reproduces the core's worked example.

The edge case. The operation destroys two numbers on purpose. Adding the same constant to every entry leaves the output unchanged. Multiplying the vector by a positive constant leaves it unchanged too. Two different inputs therefore produce one output, and the normalisation as the core defines it cannot pass the input's offset or its overall scale to the next layer. What it writes in their place is a fixed zero and a fixed one, so the offset and the scale stop here.

The order. The core's equation is the post-norm block. The vision transformer applies the normalisation before each block and the residual connection after it [S:229:3]. The two placements are different functions, and the difference is what the residual path carries. In the pre-norm version the unnormalised input travels to the top without passing a normalisation, which gives the gradient a short path. The core's claim that deep stacks remain trainable is true of residual connections. How deep a stack you can train depends on the placement.

What the core hid. It said most parameters sit in the feedforward layers. The fourfold inner width makes that true at the paper's widths, and a FLOP count for one layer shows the same split on the compute side: the feedforward part contributes 16 of the layer's 24 width-squared units of work [S:640:39].

Where the analogy breaks. The residual stream is a picture of a shared channel. The picture breaks because the channel is renormalised at every block, so its scale is not conserved. Each sub-layer writes a correction whose size its own weights set, and nothing passes through unchanged.

Original sources. The encoder layer is defined as two sub-layers, each with a residual connection followed by layer normalisation, written as the normalisation of the input plus the sub-layer's output [S:217:3]. The paper fixes six layers, a model width of 512, and an inner feedforward width of 2048 [S:217:3][S:217:5]. Prince writes the same block as four assignments, with the normalisation applied to each position vector separately [S:74:230].

An open question. Why does a stack of these blocks train at all, and how much does the placement of the normalisation decide? The 2017 block normalises after the residual addition [S:217:3]. The vision transformer normalises before each block and adds the residual afterwards [S:229:3]. The two placements are different functions, and the corpus reports no controlled comparison of the depth each reaches. Evidence: identical widths and depths trained with the two placements, without warmup and without gradient clipping, reporting the depth at which the loss diverges; then repeated with warmup to separate the optimiser from the architecture.

A frontier result. The Megatron-LM appendix counts the arithmetic of one layer and shows where the block's cost sits. The forward pass costs 24 units of width-squared work per layer, and the feedforward part contributes 16 of them [S:640:39]. A separate term grows with the square of the sequence length, which is the attention table. The same appendix gives the total for a full model, including the backward pass and the recomputation that the backward pass needs. The count is written per block, and the block is the unit that large training runs are planned in.

What to read next. Read §3.1 to §3.4 of the transformer paper for the block, the sub-layer widths and the positional signal [S:217:3]. Read Prince §12.4 for the four-line version [S:74:230]. Read the vision transformer's §3.1 for the other normalisation order [S:229:3], then the Megatron-LM appendix for the cost split [S:640:39].