Bits and floats

Write 0.1 into a 32-bit float, the number format that most machine-learning software uses for ordinary decimals and that stores a number in a fixed thirty-two bits. Read it back, and the machine returns 0.10000000149011612, a number that nobody in the program asked for. The decimal you typed and the number the machine kept differ in the ninth decimal place, and every later sum inherits that difference [S:66:72].

How is a decimal number stored in thirty-two bits? The machine keeps a sign, an exponent and a short block of binary digits, then rebuilds the number as a product [S:114:220]. The sign says positive or negative, the exponent says which power of two applies, and the digits say where inside that power the number sits. That is also the answer to the chapter's question, because a computer gets simple sums wrong when the numbers it adds are already approximations of the ones you wrote. !One word, three fields. The sign bit says positive or negative, the eight exponent bits carry the scale with a bias of 127, and the twenty-three fraction bits name a position inside that scale. The bits drawn are 0.1: 00111101110011001100110011001101, exponent field 123, so the power of two is $2^{-4}$, and the fraction reads 5,033,165 under a hidden leading one. A wider exponent buys range and a wider fraction buys precision, and the split of the bits decides both at once. Status: re-simulated.

A floating-point number is binary scientific notation with a fixed number of digits: it stores a window of digits and a power of two that slides the window along the number line. The 32-bit format splits its bits three ways: one sign bit, eight bits of exponent field, and twenty-three bits of fraction [S:105:66]. The exponent field records the power of two, and the fraction records where inside that power the number sits. The sign is a separate bit rather than a minus sign on the digits, so negative 0.1 differs from positive 0.1 in the first bit and nowhere else [S:114:220]. !The number you typed is not one the machine has. The three dots are the representable numbers around one tenth, $7.45\times 10^{-9}$ apart, and the stored word sits a fifth of a step above the decimal on the page. Reading it back gives 0.10000000149011612. The error is a quarter of one rounding, which is why the twelfth digit of the printed answer is decoration. Status: re-simulated.

The exponent field carries an offset called the bias, so you read the field as an unsigned integer, a plain binary count with no sign, and subtract 127 to get the power of two [S:114:220]. A signed integer field would work in principle, but the hardware would become more complicated and slower, so the format shifts the range by a constant instead [S:114:220]. The fraction hides its leading digit, the hidden bit: every normal number begins with the same one, so storing it would waste a bit. The twenty-three stored bits therefore act as twenty-four, the length of the significand with the leading one counted [S:114:220].

Worked example. Decode the 32-bit word, the four bytes the machine stores for one number, whose bits are 0 01111011 10011001100110011001101, the encoding of 0.1. Each field is a binary integer, so you read the eight exponent bits and the twenty-three fraction bits as ordinary whole numbers.

The sign is zero, so the number is positive, and the exponent field is 123, so subtracting the bias gives 123 − 127 = −4. The fraction, read as an integer, is 5,033,165. Adding the hidden one means adding $2^{23}=8,388,608$, because the hidden one sits at the top of a twenty-four-bit significand and the twenty-three fraction bits hang below it. The significand is therefore 8,388,608 + 5,033,165 = 13,421,773, and the power of two is $2^{-4-23}=2^{-27}$, because the stored fraction already accounts for twenty-three of the binary places.

The value is $13,421,773\times 2^{-27}$, which is 0.10000000149011612 and not one tenth. One tenth is 0.1, so the stored number is high by $1.49\times 10^{-9}$, a relative error, the size of the miss divided by the number you asked for, of $1.49\times 10^{-8}$. That misses by a quarter of the unit roundoff $2^{-24}=5.96\times 10^{-8}$, which is the largest relative error one rounding can cause. The unit roundoff is half a step at 1.0, where the step is the gap up to the next number, so its value here is $2^{-23}/2=2^{-24}$.

$$\text{value}=2^{\text{exponent field}-127}\times \left(1+\frac{\text{fraction}}{2^{23}}\right)$$

That formula is the decode written once: the bias undoes the offset, the hidden one supplies the 8,388,608, and the fraction field, read as an integer, supplies the rest. [F01]

The rule repeats with different numbers in every width. The 16-bit format keeps five exponent bits, ten fraction bits and an offset of fifteen, so it reaches a much smaller range but resolves the same kind of step [S:99:125]. Its neighbour bfloat16 keeps the eight exponent bits of the 32-bit format and only seven fraction bits, which is the same trade taken in the other direction [S:105:66]. The 64-bit format takes a wider fraction and an offset of 1023, which is why the same decimal survives longer when you move to a wider format [S:114:220].

The fraction is a sum of halves, quarters, eighths and so on, so it names 0.5 and 0.25 exactly and cannot name one tenth. The gap between neighbouring representable numbers is the format's spacing, and that gap grows with the number itself. Range is the span of magnitudes from the smallest to the largest, and precision is how finely the format divides that span. The two are separate budgets that the split of the bits decides at once: a wider exponent field reaches larger and smaller magnitudes while a wider fraction resolves finer steps.

Moving bits from the fraction to the exponent buys range at the price of precision. One more exponent bit doubles the number of binades the format reaches, while one less fraction bit halves the number of steps inside each binade. The 16-bit format does not make that trade against the 32-bit one, since it has three fewer exponent bits and thirteen fewer fraction bits, so both its range and its precision are smaller. The trade is the one bfloat16 makes against the 16-bit format, where three fraction bits move to the exponent and the range returns to the 32-bit range at the cost of precision [S:105:67].

The spacing explains why the twelfth digit of the opening number is decoration. At a number near one the spacing is $2^{-23}$, one part in $8,388,608$, which is a little finer than one part in ten million, so the machine distinguishes about seven decimal digits and no more. Ask for more digits and the answer you get back is the nearest number the format owns, not the number you asked for. That sentence is the whole of the next two sections.

The rule has a blind spot at the bottom of its range. Below the smallest normal number, the hidden leading one disappears and the number becomes subnormal, with fewer bits of precision than the format advertises [S:99:125]. There the bound on relative error still holds, but as a bound on absolute error. A relative bound compares the error to the size of the number, an absolute bound compares it to one fixed step; the gap between neighbours stops shrinking with the number, so a small number can be all error. That corner is where the gradients of a training run, the small corrections a model applies to its weights, become too small to store, and the last section of this chapter returns to it.

The formal statement. Fix the binary32 format. A finite value is a sign, an integer significand below $2^{24}$, and an integer exponent between $-126$ and $127$, and the number is the sign times the significand times two to the power of that exponent minus twenty-three. For a normal number the significand lies between $2^{23}$ and $2^{24}$, so it splits into the hidden leading one and the stored fraction. An exponent field of zero marks the subnormal case: the exponent is $-126$, the hidden one disappears, and the value is the fraction times $2^{-149}$. One rounding of a real number to this set moves it by at most half the distance to the nearest neighbour.

The two constants are the smallest positive subnormal, $2^{-149}$, and the largest finite value, $(2^{24}-1)\cdot 2^{104}$.

The derivation. Take a real number in the interval from $2^e$ to $2^{e+1}$, where $e$ names the exponent of the binade. The representable numbers inside that interval sit $2^{e-23}$ apart, because the fraction holds twenty-three bits below the hidden one. Rounding to the nearer of two neighbours moves the value by at most half that gap. Divide by the value, which is at least $2^e$, and the relative error is at most $2^{-24}$. That number is the unit roundoff. It is a relative bound, so it holds at every scale where the numbers are normal.

$$\text{spacing at }x=2^{\lfloor {\log }_{2}x\rfloor -23}$$

That line says: the gap between neighbouring values is a power of two set by the binade the number sits in. The bound needs rounding to nearest: any nearest rule moves a value by at most half a step, so $|\delta |\le u$. Ties-to-even is a separate requirement, and it is the one that keeps the rule unbiased; without it, a tie rule that always rounds up biases every result in the same direction [S:28:49].

I omit the proof that the format holds no other values. It is a count of bit patterns and adds nothing here.

The edge case. The range of the format is a property of stored values, not of intermediate ones. A product can overflow while the mathematical answer is representable: computing a vector norm by squaring each entry overflows when an entry is near the largest value, even though the norm itself is inside the range [S:28:55]. The repair is the same shift you meet in 6.3, dividing by the largest entry first. A reader who treats the range as a statement about arithmetic will miss this, because no operand leaves it.

What the core hid. The spacing is constant inside each binade and doubles at every power of two, so "the gap grows with the number" describes a staircase rather than a slope. The core also said the format keeps about seven decimal digits. That is the worst case, not a promise for a particular result: the format has twenty-four binary digits, and the decimal count varies with where the number sits. Nothing in the format keeps a digit you did not ask for.

Where the analogy breaks. The core called the format binary scientific notation. Scientific notation keeps every digit you write and moves the point. The format keeps a fixed width and discards what does not fit. It also spends one exponent pattern, all ones, on the special values: with that pattern and a zero fraction the value is infinity, and with a nonzero fraction it is NaN, so the format is not a notation for the reals [S:114:220]. It is a finite set with holes.

Original sources. The layout is fixed by IEEE 754, and the CUDA programming guide names the revision its devices follow, IEEE 754-2008 [S:3:506]. The textbook account of the three fields and the hidden bit is section 5.3 of OpenStax [S:114:219]. The rounding rule that produces the unit roundoff, round to nearest with ties to even, is in section 2.1 of Solomon [S:28:49]. The parameters themselves belong to the standard. Solomon states the double-precision case and OpenStax the single-precision split.

An open question. Whether a training stack may rely on gradual underflow is unsettled. The standard defines subnormal values, and the CUDA compiler exposes a flush-to-zero mode (-ftz=true) that discards them. The guide requires -ftz=false for IEEE compliance, and single-precision atomic adds on global memory flush to zero whatever the flag says [S:3:506]. The exception runs the other way for one intrinsic: the fast tanhf does not flush its subnormal results even under -ftz=true [S:3:432]. Two devices can therefore return different answers for the same kernel on the same input, in exactly the range where small gradients and small probabilities live. Evidence that would settle it: one kernel, compiled for each device, run on inputs whose results land in the subnormal range, with the outputs compared bit for bit. The comparison has to fix the flush mode as well as the format, because the mode is a property of the compilation as well as of the device.

A frontier result. The narrow formats now arriving are built by moving the split, not by shrinking the whole word. bfloat16 keeps the range of float32 and pays for it with fewer fraction bits [S:105:67]. NVFP4 keeps one sign bit, two exponent bits and one fraction bit, and buys range back with a shared scale for each block of sixteen values [S:408:2]. Both are the trade the core described, made again at a smaller word.

What to read next. Read section 5.3 of OpenStax for the field layout and the subnormal exception [S:114:219]. Read section 2.1 of Solomon for machine precision and the rounding rule [S:28:49]. Read the half-precision table in the NVIDIA guide for the same split at sixteen bits [S:99:125], and then the NVFP4 report for the split at four [S:408:2].