What the model learned

Inside GPT-3, one token at one layer is a list of 12,288 numbers. The model has 96 layers, so a single token passes through more than a million numbers on its way to a prediction [S:232:8]. Nobody wrote those numbers; training set them.

A hidden layer is one of the intermediate stages that turns the input into the output. Its representation of a token is the vector of numbers that stage hands to the next one [S:91:11]. The answer to this section's question is therefore concrete: a hidden layer contains one vector per token, and each coordinate of that vector is a learned number. !One unit, then a layer. A token with coordinates 3 and 4 enters two units: weights $(2,-1)$ with bias 0.5 return $2(3)-1(4)+0.5=2.5$, and weights $(-1,2)$ with bias 0 return 5. A layer stacks 12,288 such units, so its output is a vector with one coordinate per unit. The figure is schematic, drawn from the section's worked example. Status: schematic.

The first stage maps a token index to one fixed vector, the token's embedding [S:91:11]. That vector is the same wherever the word appears. Later layers rewrite it using the surrounding tokens, so the vector deep in the stack differs from the embedding at the input. !Meaning as a direction. In the reported analogy, the vector for king minus man plus woman lands closest to queen, so the offset from man to woman is the same as the offset from king to queen. The figure is a schematic of that claim, not a measured embedding; the axes have no names, and only directions and relative positions carry meaning. Status: schematic.

A transformer block has two sublayers. Multi-head attention makes each token's output a weighted sum of the other positions' vectors, and a position-wise MLP transforms each position on its own [S:232:8-9; S:313:6-7]. Attention is the part that lets a word's representation depend on its sentence. A hidden layer is useful because it can rewrite that vector before the output decision, so the output layer sees a vector shaped for the task rather than the raw token identity.

Each coordinate in the MLP comes from a small calculation. A unit takes the input vector, multiplies each input by a weight, adds the results, and adds a bias. For an input with two coordinates,

$$h=w_1{x}_1+w_2{x}_2+b.$$

That equation says a unit is a weighted sum plus an offset, and the weights $w$ decide which input coordinates $x$ matter to it. The result is one coordinate $h$ of the next layer, and the bias $b$ shifts it up or down. A layer stacks many units, so its output vector has one coordinate per unit. A hidden layer therefore holds two kinds of learned numbers: weights that connect coordinates, and biases that shift each unit.

Worked example. A token arrives with coordinates 3 and 4. One unit has weights 2 and -1 and bias 0.5, so it returns $2(3)+(-1)(4)+0.5=2.5$.

A second unit has weights -1 and 2 and bias 0, so it returns $(-1)(3)+2(4)=5$. The layer's output for this token is the pair 2.5 and 5. A dense layer with $m$ units reading $n$ coordinates has $mn+m$ parameters: linear in each dimension, and quadratic only when the layer is square. Widening a square layer costs $m^2$, and adding another layer costs another $m^2$. The width-depth choice is settled by compute, latency and optimisation behaviour, not by this arithmetic alone.

The arithmetic therefore explains why increasing width is expensive. A layer with 12,288 units, each reading the same 12,288-number input, holds about 151 million weights in one square projection matrix. A transformer block holds four such attention projections and a feed-forward network four times as wide, so the block is far larger than that one matrix. Ninety-six blocks account for the whole model's parameter count [S:232:8-9; S:313:6-7].

The numbers are not a code with named slots; they are coordinates in a space, and training arranges that space so that useful distinctions become directions. The demonstration is that the vector for king, minus the vector for man, plus the vector for woman, lands closest to queen [S:184:2]. That result is about static word2vec vectors, one per word type, scored by nearest neighbours on an analogy set.

It does not transfer as stated to a transformer's hidden states, which are contextual. It supports the claim that meaning is a direction in a learned embedding space, not that a hidden layer computes king minus man plus woman. The geometry is real; the arithmetic demonstration is narrower than it looks.

The distinction between a coordinate and a concept matters for reading a model. A coordinate with a large weight is not necessarily a concept a person would name, and a concept may be spread across many coordinates; the geometry is real, but the labels are ours.

One way to test what a vector carries is a probe: a small classifier, a model that assigns a label. Train it on the vector and see whether a property, such as part of speech, can be read out [S:446:5]. A probe that succeeds shows the information is present, not that the model uses it.

The idea breaks when we read a direction as one clean concept. Some features are circular: GPT-2 represents days of the week and months of the year as circles in a two-dimensional subspace, a flat slice of the space, rather than as independent lines [S:437:1]. Other features may need several dimensions at once, so a single direction can mix concepts [S:437:1-2]; what the model learned is real, but it is not a dictionary.

The formal statement. A layer with $m$ units reads an input vector and returns $m$ numbers. Each unit computes a weighted sum of the input coordinates plus its own bias, and the layer returns the list of those sums, one per unit. Writing the weights as rows of a table and the input as a column, the whole layer is one matrix-vector product followed by a function applied to each coordinate. The matrix has one row per unit and one column per input coordinate, so its shape is $m$ by $n$, and the layer holds $mn+m$ parameters once the biases are counted. This is the position-wise MLP of a transformer block. Attention is a separate operation: it mixes information across token positions, which the MLP cannot do.

The derivation. The matrix form is bookkeeping on top of L1's arithmetic. Each row of the weight table holds the weights of one unit, and the matching entry of the product is the dot product of that row with the input. Adding the bias vector gives the value that the unit passes on. Nothing is approximated: the layer computes exactly the sum L1 wrote, once per unit. The count $mn+m$ is linear in each dimension and quadratic only when $m=n$.

The step I omit is the proof that a stack of linear layers without a nonlinearity collapses to a single linear layer. The identity behind it is that applying one matrix and then another is applying their product, and that is why the nonlinearity is not decoration.

An edge case. L1 says each coordinate of a hidden vector is a learned number. That is true, and it invites a wrong reading, that each coordinate is one concept. A single unit responds to several unrelated kinds of input, a property called polysemanticity [S:446:8]. A model can also represent more features than it has coordinates by keeping each feature sparse, which is the superposition hypothesis [S:446:9].

A coordinate can therefore carry several features, and one feature can be spread across several coordinates.

What L1 hid, and why that was safe there. The worked example in L1 adds two weighted sums and stops. It omits the function applied afterwards, so the layer it describes is linear. With 12,288 coordinates and 96 layers, a purely linear stack would be no more expressive than one matrix [S:232:8]. The omission is safe for the arithmetic lesson, because the weights and the bias are the same either way.

It becomes unsafe if a reader concludes that depth by itself buys expressive power.

The block count follows from the same arithmetic. A GPT-3 block holds four attention projections of $12,288^2$ weights, about 604 million, and an MLP of $12,288\to 49,152\to 12,288$, about 1.2 billion, so a block holds about 1.8 billion parameters. Ninety-six blocks give about 174 billion, which is the 175 billion total once embeddings and biases are counted [S:232:8-9; S:313:6-7].

Where the analogy breaks. "A learned coordinate system" suggests axes with fixed meanings, like latitude and longitude. The axes are not fixed. The same token receives different vectors in different sentences, and one direction can serve several features. L1 says this at the end; the formal version drops the metaphor of axes and keeps only the vectors.

Original sources. The demonstration that word vectors support arithmetic is Mikolov and coauthors'. The vector for king, minus the vector for man, plus the vector for woman, lands closest to the vector for queen [S:184:2]. The description of a hidden state as a sequence of real-valued vectors is standard in the pretraining literature [S:91:11]. The claim that concepts map to directions in activation space is the linear representation hypothesis [S:446:10].

An open question. The field has not settled whether a feature is always a single direction. Some features are circular: days of the week and months of the year are laid out on a circle rather than on a line [S:437:1]. The authors test that hypothesis and call its one-dimensional part into question [S:437:1-2].

The evidence that would settle a given feature is an intervention. Change the candidate representation and check whether the model's behaviour changes in the predicted direction, then repeat the test on a feature that the method was not built to find.

A frontier method. Sparse autoencoders learn a wide, sparse layer that reconstructs a model's internal activations, so each active unit becomes a candidate feature [S:439:1]. The method depends on this section because it assumes that features are directions in activation space which a dictionary can recover. If some features are curved, a dictionary of straight directions will split them or miss them. That is the open question above, stated as a design assumption.

What to read next. Read Mikolov et al. for the arithmetic demonstration and what it was taken to show [S:184:2], then the circular-features paper for the counterexample [S:437:1]. Read the sparse-autoencoder paper for how features are extracted at scale [S:439:1]. Read the probe section of the open-problems review, which explains why a probe that succeeds is not evidence that the model uses the feature [S:446:20].