Deeper

Lab · c24.s03.L1

The scaling-law curve

Scaling is a power law with diminishing returns: every tenfold increase in compute multiplies the removable loss by the same factor, 0.708, however far the curve runs.

Question: What happens when you multiply data, parameters and compute?

The scaling-law curveA log–log plot on paper: one navy curve falls from removable loss 1.336 at 10^18 FLOPs to 0.119 at 10^25. A russet marker at 1.00 × 10^22 reads 9.129B parameters, 182.6B tokens and removable loss 0.336 of total 2.04. Gold rungs mark each decade; every one multiplies the removable part by 0.708, a 29.2% cut, while counts grow ×3.162. A second view adds the fixed 1.70 floor.The scaling-law curveloss against compute, 20 tokens per parameter, one fixed floorNow: 1.00 × 10^22 FLOPs9.129B parameters · 182.6B tokensremovable loss 0.336 of total 2.04Each decade does the same1.00 × 10^22 → 1.00 × 10^23removable 0.336 → 0.238 · ×0.708removable loss above the floor (nats/token), log0.10.20.51.010^1810^1910^2010^2110^2210^2310^2410^25compute (FLOPs), log scale0.336gold rungs: removable loss at each decade · every ×10 step leaves ×0.708 of the removable partparameters and tokens grow ×3.162 per decade: compute = 6 × parameters × tokensSynthetic power law: exponents −0.30, E = 1.70, D = 20N, C = 6ND · constants illustrative, not a fit
Synthetic Synthetic: the curve comes from the stated power law with rounded exponents and C = 6 N D, D = 20 N, so no point is a measurement and the constants are not a published fit. At 1.00 × 10^22 FLOPs the total loss is 2.04 with 0.336 removable, 9.129B parameters and 182.6B training tokens; at 1.00 × 10^23 it is 1.94 with 0.238 removable, 28.87B parameters and 577.4B tokens. Every decade multiplies the removable loss by 0.708, a 29.2% cut, while the counts grow ×3.162.
Lab controls
22

Left/Down and Right/Up move one quarter-decade, a factor of ×1.778 per step; Page Down/Page Up move a larger step; Home sets 10^18; End sets 10^25

Above the floor

Left/Right move between the two options; Space or Enter keeps the focused choice; the chosen view is announced as text

On

Space or Enter toggles the ladder; the control reports On or Off as text, never colour alone

Frame 1 of 1: 10^22 FLOPs, above the floor1.00 × 10^22 FLOPs: 9.129B parameters and 182.6B training tokens. Removable loss is 0.336 of the total 2.04; the fixed floor 1.70 is subtracted on this view. One decade on, at 1.00 × 10^23, the removable loss is 0.238, a factor of 0.708 and a 0.098 drop in total loss. Gold rungs mark the removable loss at every decade; the marker sits at the current budget.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A log–log plot on paper: one navy curve falls from removable loss 1.336 at 10^18 FLOPs to 0.119 at 10^25. A russet marker at 1.00 × 10^22 reads 9.129B parameters, 182.6B tokens and removable loss 0.336 of total 2.04. Gold rungs mark each decade; every one multiplies the removable part by 0.708, a 29.2% cut, while counts grow ×3.162. A second view adds the fixed 1.70 floor.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.