Practise: The Real Machine

The sign bit is 0, so the number is positive. The exponent field is $$10000001_2=128+1=129,$$ and subtracting the bias gives $129-127=2$, so the scale is $2^2=4$. The fraction reads as an integer and divides by $2^{23}$: $$\frac{01000000000000000000000_2}{2^{23}}=\frac{2^{21}}{2^{23}}=\frac{1}{4}.$$ Adding the hidden leading one gives $1+\frac{1}{4}=\frac{5}{4}$. The value is therefore $$4\times \frac{5}{4}=5.$$ Encoding 5.0 again returns the same word, which is the check that the decode lost nothing. Section 6.1's formula is the whole method: subtract the bias, build a number between one and two from the fraction, multiply.

The number of decimal digits that a binary significand of $b$ bits can distinguish is $b{\log }_{10}2$, because each bit multiplies the count of distinguishable values by two and each decimal digit multiplies it by ten. With $b=24$, $$d=24{\log }_{10}2=24\times 0.3010299957\dots =7.2247\dots ,$$ which rounds to 7.22. The format therefore separates about seven decimal digits near one, and section 6.1's remark that the twelfth digit of the opening number is decoration follows. The count is a rate, not a guarantee at every magnitude: at a number near $10^8$ the spacing has grown to 8, so only the digits above the units place survive. Seven is the budget at the bottom of the binade, where the spacing is smallest relative to the number.

The subtraction gives $$3.141593-3.141592=0.000001=10^{-6}.$$ The cancellation ratio is the larger operand divided by the answer: $$\frac{3.141593}{10^{-6}}=3.141593\times 10^6\approx 3.14\times 10^6.$$ The ratio says what the subtraction does to the error already present. Both operands sit in the binade $[2,4)$, where the 32-bit spacing is $2^{-22}=2.38\times 10^{-7}$ and half a step is $1.19\times 10^{-7}$. Two operands can disagree by twice that, $2.38\times 10^{-7}$, so the difference of $10^{-6}$ can be wrong by about 24 per cent of itself. The digits were lost in the earlier arithmetic that produced the measurements, and the subtraction only brought the loss into view.

The exact sum is $33,554,435$. Representable numbers in this binade are multiples of 4, so the machine keeps the nearest one. The neighbours are $33,554,432$ and $33,554,436$, and the second is nearer, at distance 1 rather than 3. The stored sum is therefore $33,554,436$, and the subtraction is exact: $$33,554,436-33,554,432=4.$$ Against the true difference 3 the relative error is $$\frac{4-3}{3}=\frac{1}{3}\approx 0.333,$$ or 33.3 per cent. Both operations were exact once the sum was stored; the digit went missing in the addition, as section 6.2 says. A wider subtraction would return the same 4, because it would subtract the same stored sum.

The largest logit is $\ln 2$. Subtracting it from each logit gives $$-\ln 2,-\ln 2,0.$$ Exponentiating gives $\frac{1}{2}$, $\frac{1}{2}$ and $1$, whose sum is 2. The probabilities are therefore $$\frac{1/2}{2}=\frac{1}{4},\frac{1/2}{2}=\frac{1}{4},\frac{1}{2}=\frac{1}{2}.$$ The answer is $0.25$, $0.25$ and $0.5$. The shift changed every intermediate number and left every ratio alone, which is the algebra section 6.3 relies on. The largest probability is exactly one half because the third logit is one natural logarithm above the other two, a factor of two in the exponentials.

The three byte counts are $$mk=2048\times 512=1,048,576,bk=256\times 512=131,072,bm=256\times 2048=524,288.$$ Their sum is $1,703,936$, and doubling for two bytes per number gives $$2\times 1,703,936=3,407,872\text{bytes}.$$ The operation count is $$2bmk=2\times 256\times 2048\times 512=536,870,912.$$ The intensity is the ratio: $$\frac{536,870,912}{3,407,872}=157.538\dots \approx \mathrm{157.54.}$$ That value lies above the main-memory ridge of 139, so the layer is compute-bound: the arithmetic takes longer than the traffic. The margin is thin, about 13 per cent. Moving the same layer to a cache with a ridge of 40 would leave it compute-bound by a wide margin, while a batch ten times smaller would move it below the ridge.

The accounting identity of section 6.5 says that the bytes in flight equal the rate multiplied by the wait: $$600\times 10^9\text{bytes per second}\times 400\times 10^{-9}\text{seconds}=240,000\text{bytes}.$$ Each request carries 32 bytes, so the number of requests is $$\frac{240,000}{32}=7,500.$$ One request at a time would deliver $32/400\times 10^{-9}=80$ million bytes a second, and the ratio of 600 billion to 80 million is again 7,500. The threads are the currency because each waiting thread keeps one request alive. Doubling the wait would double the count, which is why a slower memory demands more parallelism rather than a faster core.

The gradient must satisfy $$s\times 1.5\times 10^{-8}\ge 6.1035\times 10^{-5},$$ so the scale must be at least $$s\ge \frac{6.1035\times 10^{-5}}{1.5\times 10^{-8}}=\mathrm{4069.01.}$$ Loss scaling multiplies by a power of two, because that multiplication only moves the exponent and leaves the significand alone. The powers of two around the threshold are $2^{11}=2048$ and $2^{12}=4096$, so the smallest admissible scale is 4096. The check is $$4096\times 1.5\times 10^{-8}=6.144\times 10^{-5},$$ which is above $6.1035\times 10^{-5}$ and therefore normal. The scale 2048 gives $3.072\times 10^{-5}$, a subnormal with fewer than ten bits of precision. Section 6.6's distinction holds here: the scale buys range, and the precision of the scaled gradient is whatever the format gives at that magnitude.