Derive: Scale, Language Models and What They Learned

Write the exponent as $$H=-\frac{1}{T}\sum _{t=1}^T{\log }_2{p}_t={\log }_2\left[{\left(\prod _{t=1}^T{p}_t\right)}^{-1/T}\right].$$ Exponentiating base 2 gives $$\mathrm{PP}=2^H={\left(\prod _{t=1}^T{p}_t\right)}^{-1/T}.$$ For $T=2$, the product is $0.5\times 0.01=0.005$, so $$\mathrm{PP}=0.005^{-1/2}=\frac{1}{\sqrt{0.005}}=\mathrm{14.142.}$$ The per-token values are $1/0.5=2$ and $1/0.01=100$, whose arithmetic mean is 51. Perplexity is built from an average of logarithms, and exponentiating an average of logarithms gives a geometric mean, not an arithmetic one. The geometric mean is pulled toward the small value; the arithmetic mean is pulled toward the large one. This derives the bridge between the sequence product of section 24.1 and the average surprise of section 24.2.

Substituting the rule gives $$C=6N(20N)=120N^2.$$ With $C=10^{23}$, $$N=\sqrt{\frac{10^{23}}{120}}=\sqrt{8.3333\times 10^{20}}=2.88675\times 10^{10}.$$ That is 28.9 billion parameters to one decimal place. The token count is $D=20N=5.7735\times 10^{11}$, about 577 billion tokens. The check is $6\times 2.88675\times 10^{10}\times 5.7735\times 10^{11}=10^{23}$. The rule ties the two choices: a larger model demands proportionally more tokens, so one budget fixes both. If the rule were 40 tokens per parameter, the budget would be $C=240N^2$, and $N$ would fall by $\sqrt{2}$ to 20.4 billion while $D$ rose to 816 billion. The closed form makes the trade visible; the exponent in the loss law decides whether the trade is worth making.

In the order $A$ then $B$, $A$ receives $0.4-0.1=0.3$ and $B$ receives $0.9-0.4=0.5$. In the order $B$ then $A$, $B$ receives $0.3-0.1=0.2$ and $A$ receives $0.9-0.3=0.6$. Averaging the two orders gives $${\varphi }_A=\frac{0.3+0.6}{2}=0.45,{\varphi }_B=\frac{0.5+0.2}{2}=\mathrm{0.35.}$$ Their sum is 0.80, which equals $f(\{A,B\})-f(\varnothing )=0.9-0.1$. That is the additivity guarantee in this small case. Now remove each feature from the full set on its own. Removing $A$ leaves $f(\{B\})=0.3$, a drop of 0.6. Removing $B$ leaves $f(\{A\})=0.4$, a drop of 0.5. The two drops sum to 1.1, which exceeds the joint drop 0.8 by 0.3. The Shapley average splits the interaction across the two orders; the one-at-a-time table charges the interaction twice. This is why the additivity guarantee in section 24.5 is a property of a stated division rule, not of every removal experiment.