Practise: Scale, Language Models and What They Learned

The second tokeniser leaves $1,024-800=224$ positions unused. The fraction is $224/1,024=0.21875$, which rounds to 0.219. The first tokeniser would leave $384/1,024=0.375$ unused. The passage did not change; the vocabulary did. A token is not a word, so the count that matters is positions in the window, not words on the page.

The first two factors give $0.5\times 0.25=0.125$. Multiplying by the third gives $0.125\times 0.8=0.1$. Each factor is conditioned on all earlier tokens, so the model's score for the sequence falls as the sequence grows even when every single step is likely. A fourth token at probability 0.8 would take the score to 0.08. The number is the model's probability for this sequence, not a claim that the sequence is true or useful.

The surprises are $-{\log }_{2}0.5=1$, $-{\log }_{2}0.125=3$, and $-{\log }_{2}0.8=0.321928$ bits. Their average is $(1+3+0.321928)/3=1.440643$ bits. Perplexity is $2^{1.440643}=2.714$ to three decimal places. The same number comes from the product: $0.5\times 0.125\times 0.8=0.05$, and $0.05^{-1/3}=2.714$. A perplexity of 2.714 means the model's uncertainty per token matches a uniform choice among about 2.714 tokens. It is not a probability, and it can exceed 1.

The multiplier is $100^{-0.06}=10^{-0.12}=0.75858$. The new loss is 75.858 percent of the old loss, so the fall is $1-0.75858=0.24142$, or 24.1 percent. A thousandfold increase would give $1,000^{-0.06}=10^{-0.18}=0.66069$, a fall of 33.9 percent. Each extra factor of ten buys less than the one before. That is what a small exponent means: the curve keeps falling, but the steps shrink.

The weighted sum is $1+(-4)+(-6)=-9$. Adding the bias gives $-9+1.5=-7.5$. The unit is a weighted sum plus an offset; the bias moves the result after the inputs have voted. If the third input changed from 3 to $-3$, the third product would become $+6$ instead of $-6$, and the unit would return 4.5.

One unit holds $4,096$ weights and one bias, so $4,097$ numbers. The layer holds $4,096\times 4,097=16,781,312$ numbers, because $4,096^2=16,777,216$ and adding $4,096$ gives the rest. That is about 16.8 million numbers in one layer. The chapter's 12,288-wide layer holds about 151 million. Width costs quadratically because every unit reads every coordinate.

Removing great drops the prediction by $0.8-0.45=0.35$. Removing film drops it by $0.8-0.72=0.08$. The two single drops sum to 0.43. Removing both drops the prediction by $0.8-0.30=0.50$. The interaction term is $0.50-0.43=0.07$. The two features together remove more than the sum of their separate removals, so a table of single-feature attributions does not add up to the joint effect. The guarantee attached to SHAP belongs to a defined division rule, not to any table built by removing one feature at a time.

The gaps are $0.8-0.2=0.6$, $0.6-0.9=-0.3$, and $0.7-0.5=0.2$. The largest is 0.6, on the first answer. The proxy ranks that answer top and the humans rank it bottom. The gap measures the distance between two scoring rules on these three items, not a general error rate for the model. A negative gap means the proxy undervalues an answer the humans prefer.