Mixed precision

A training run adjusts a model's weights over many steps. One training step has two halves. A forward pass computes a loss, the number the model is trying to shrink, and a backward pass computes one gradient per weight, the direction and size of the change that would reduce the loss.

Three of those gradients are 0.00000001, 0.00000003 and 0.0000001. Store them in the 16-bit format and retrieve them, and the first becomes zero, the second becomes $5.96\times 10^{-8}$, the nearest nonzero value the format holds, and the third becomes $1.19\times 10^{-7}$ [S:99:125]. Nothing reports a failure, and the weight attached to the first gradient does not move at this step; it moves again only when a later gradient is large enough to survive the format. !Half precision near zero is a grid that stops before the gradients begin. Its smallest subnormal is $5.96\times 10^{-8}$, so the first rounding boundary above zero is half of that, $2.98\times 10^{-8}$. The gradient 0.00000001 lies below the boundary and becomes zero, 0.00000003 rounds up to one step, and 0.0000001 rounds to two. Scaling by a power of two slides the values along this axis and leaves the grid where it is: a scale of 8 buys survival, and 8192 buys accuracy. Status: re-simulated.

When can you use fewer bits on purpose? When every value stays inside the format's range, and every change you care about is larger than the format's step, the spacing between neighbouring stored values, which are two separate budgets that a format buys one against the other [S:99:125].

Half precision is the 16-bit format with five exponent bits and ten fraction bits; its largest finite value is 65,504 and its smallest normal value is $2^{-14}$ [S:99:125]. Mixed precision means training with more than one format at once, narrow where the traffic dominates and wide where the accumulation happens, in the long sums that build a result [S:105:68]. Loss scaling multiplies the loss by a power of two before the backward pass so that the gradients land inside the narrow format's range. It divides them back before the update, the change the step makes to each weight [S:105:68]. A master copy is a higher-precision copy of the weights, and swamping is the loss of an update that is smaller than the step of the weight it changes.

The two budgets fail in different directions, because a value outside the range becomes zero or infinity while a value inside the range can still be too coarse to change anything. Scaling by a power of two leaves the relative precision of a normal value unchanged: it moves the exponent and the fraction is untouched. A subnormal value is different, because it carries fewer significand bits the smaller its exponent, so scaling it into the normal range restores precision. That is why a gradient of $10^{-8}$, which the 16-bit format stores as zero, keeps eleven bits at a scale of 8192, and why the scale repairs the range and the subnormal precision but not the step at a normal weight.

$$|\text{scaled gradient}|>\frac{1}{2}\times \text{smallest subnormal}$$

That line is the survival condition: a scaled gradient above half the smallest subnormal rounds to a nonzero value, and at exactly half the tie goes to zero. [F01]

Worked example. Five exponent bits put the smallest normal number at $2^{-14}$, and ten fraction bits put the step below it at $2^{-24}$, so the smallest subnormal of the 16-bit format is $5.96\times 10^{-8}$ [S:99:125]. Round to nearest sends anything below half a step to zero, so the first boundary above zero is half of that, $2.98\times 10^{-8}$. The gradient 0.00000001 lies below the boundary and rounds to zero.

The gradient 0.00000003 lies above it and rounds to one step, $5.96\times 10^{-8}$, which is wrong by 98.7 per cent of itself because the step is nearly as large as the value it stores. The gradient 0.0000001 rounds to two steps, $1.19\times 10^{-7}$, which is wrong by 19.2 per cent.

Now scale by 8, and the first gradient becomes 0.00000008, which lands on one step, $5.96\times 10^{-8}$, and is wrong by 25.5 per cent because that step is about three quarters of the value. Scale by 8192 instead, and it becomes 0.00008192, above the subnormal range, so all eleven significand bits are back. The scale bought survival at 8 and accuracy at 8192, and those are different purchases at different prices.

The second place where width matters is a long sum, because every addition rounds and the roundings accumulate across the reduction, the sum over many terms. A long reduction can lose about one rounding per addition, so a sum in half precision needs a wider accumulator, a running total kept in a wider format, even when every term is comfortably in range [S:105:68].

The repairs consume memory and time, since a master copy in 32 bits means four bytes for every weight alongside the two-byte copy [S:105:68]. The arithmetic is also no longer the arithmetic of the wide format, so a result that differs from the reference is anticipated rather than surprising. A scale repairs the range and the subnormal precision, but it cannot add precision to a value that is already normal, and it cannot make an update below half the weight's spacing move the weight [S:99:127]. The 4-bit formats now arriving keep the same two budgets and make both of them tighter [S:408:1].

The formal statement. Fix a format with a smallest positive subnormal, a largest finite value, and a fixed number of significand bits. A real number rounds to a finite nonzero value when it is strictly above half the smallest subnormal and strictly below the overflow threshold, which for binary16 is 65,520, the midpoint between the largest finite value 65,504 and $2^{16}$. Values in $(2.98\times 10^{-8},65,520)$ survive; at $2.98\times 10^{-8}$ the tie goes to zero. A change to a stored weight $w$ survives the addition when it is at least half the spacing at $w$, because below that the sum is nearer $w$ than the next value and rounds back. The tie at exactly half a step depends on the rounding rule. One unit in the last place is at most $2^{1-p}$ times the value, where $p$ is the number of significand bits. So a change of at least $2^{-p}$ of the weight always moves it, and one below $2^{-p-1}$ of the weight never does.

$$\text{a change survives}⟺\text{change}\ge \frac{1}{2}\text{spacing at }w$$

That line is the swamping condition: a change smaller than half a step is added and then rounded away.

The derivation. The first bound is the rounding rule applied to zero: below half the smallest subnormal, the nearest representable value is zero. The second is the same rule applied to the sum. Inside a binade the spacing is a fixed step, so adding a change smaller than half the step leaves the sum nearer to the old value than to the next one. The sum rounds back and the update disappears, with no error reported. The statement is about the step at the weight, not about the precision of the gradient by itself.

The edge case. One scale must satisfy both bounds at once. Scaling every gradient by the same factor requires the scaled values to sit between half the smallest subnormal and the largest finite value at the same time. Such a factor exists only if the gradients span less than the dynamic range of the format, and half precision spans forty powers of two including the subnormals [S:99:125]. A run whose gradients span more than that cannot be rescued by any single scale. The failure then looks like a diverging loss rather than a rounded one.

What the core hid. The core treated mixed precision as one decision. It is three: the format of the stored tensors, the format of the accumulation, and the width of the master copy [S:105:68]. The three have different jobs. The opening example of the core fails in the first, and the closing paragraph repairs the third.

Where the analogy breaks. "The step of the format" suggests one step per format. The step is a function of the value and doubles at every power of two. A weight of 1.0 and a weight of 2.0 have different steps in the same format, so any single quoted step is a statement about one binade.

Original sources. The loss-scaling step is section 2.3.1 of the NVIDIA mixed-precision guide [S:99:127], and the three-step recipe, an FP32 master copy, loss scaling and FP32 accumulation, is section 3 [S:99:129]. The account that attributes the three tricks to the original paper is the Ultra-Scale playbook [S:105:68]. That paper is Micikevicius and coauthors from 2018, and the corpus holds it as an entry in a reference list rather than as a text [S:105:89].

An open question. How narrow a format training tolerates is unsettled. At four bits the range and the step fail together, and the published recipe adds random rotations, block scaling, stochastic rounding and a set of layers kept wide [S:408:1]. At 12 billion parameters and 10 trillion tokens the paper's ablation removes each component in turn and reports that every one is needed for convergence and stability, and it calls gradient stochastic rounding essential [S:408:5-6][S:408:9]. What is not known is whether the same set is necessary at other model sizes and token horizons, since the paper says smaller models trained for fewer tokens may not require all of it, and which parts pay for themselves only by extending the run. Evidence that would settle it: one ablation at a single model size and token budget, with each part removed alone and the loss compared at fixed compute rather than at a fixed step count. A component that pays for itself only by extending the run is a speed result, not a stability result.

A frontier result. A twelve-billion-parameter model trained on ten trillion tokens in a four-bit format, with loss and downstream accuracy close to an eight-bit baseline [S:408:2]. Stochastic rounding of the gradients is reported as necessary for convergence at that scale, and stochastic rounding of the forward tensors as detrimental [S:408:9]. That is the swamping argument of the formal statement, applied to the rounding rule instead of the format.

What to read next. Read section 2.3 of the NVIDIA guide for the three tricks and the dynamic scale [S:99:127][S:99:128]. Read the mixed-precision section of the Ultra-Scale playbook for the eight-bit case [S:105:68]. Then read the NVFP4 report for the four-bit case, and read its ablation section before believing that any single trick is the cause [S:408:9].