Lab · c21.s05.L1

Transformer walkthrough

One block does two jobs in a fixed order: attention mixes the six positions, and the small network rewrites one position at a time.

Question: How do attention and a small network fit together?

Schematic Schematic: one block with chosen, untrained weights; layer normalisation, dropout and multiple heads are omitted. Five steps on six tokens of the cat sat on a mat: the chosen position's four numbers, the attention shares that built them, the eight hidden numbers and both additions. Every number shown is the exact arithmetic of the fixed weights.

Lab controls

Position1

Left and Right move one position, Home goes to the first, End to the last; the word and index are announced on change

Time per step1600 ms

Left and Right change the dwell by 400 ms; the stored dwell is 1600 ms

Frame 1 of 5: the: input 1.00The block receives four numbers at position 1, "the": 1.00, 0.00, 0.50, 0.00. The same four-number vector is handed in at every position, and the steps ahead are fixed: attention, add, small network, add. No position keeps more than 0.18 of itself once attention mixes; that largest own share belongs to "on".

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

Six tokens of the cat sat on a mat, four numbers each. Five boxes name the steps: input, attention, add, small network, add. The chosen position's numbers appear as bars; attention blends all six: 0.05 of the blend is the token's own value, 0.31 arrives from cat. A shared small network rewrites them at that position through eight hidden numbers, and two additions keep earlier vectors: 1.00 becomes 1.93, then 3.96. Attention mixes positions; the network cannot.

Return to the section that hosts this lab

Caption status: schematic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.