Section 6.6 gives two budgets and three repairs. A narrow format fails when a value leaves its range, and it fails again when a change is smaller than its step. Loss scaling moves values into the range, a master copy keeps a wide version of the weights, and a wider accumulator repairs a long sum. The section does not say which tensors may go narrow, and no rule in it chooses a format per tensor.
That choice is now made thousands of times in a training run, and the field has no settled procedure for it. A single global loss scale is the common default, and it fails when the gradients of different layers sit at different magnitudes. Per-tensor scales fix that and introduce a new question: which statistic of a tensor, measured when, decides the scale. The 8-bit and 4-bit formats arriving now sharpen the question. Their range is small enough that the scale must be recomputed during the run, and a scale that changes changes the arithmetic.
Take a position on this question: what procedure should decide the format and the scale of each tensor in a training run, and what should the run be allowed to claim about its result?
In your answer, name the quantity you measure and the moment you measure it. State the rule that turns the measurement into a format and a scale. Say what the run promises about its result, against which reference and on which metric. Name one way your procedure fails even when every step is followed. State what your procedure costs in memory, traffic or time.
Rubric. Tick each criterion your answer meets, then save the self-assessment. The platform does not grade it.
Worked solution
There is no single correct answer, so this is a model of the shape a complete answer takes, not a key. A level-3 answer starts from the two budgets. Range fails at the top and the bottom, so the procedure watches the largest absolute value of a tensor over a window of steps. It sets the scale to the largest power of two that keeps that value below the format's largest finite number with a margin. It sets that scale per tensor rather than per run. Section 6.6's swamping failure is a statement about one weight's step, and a global scale chosen for the largest gradient in the model leaves the smallest gradients subnormal. Step fails at the weight, so the procedure also watches the ratio of the update to the step at the weight. It keeps a master copy in 32 bits whenever that ratio falls below one at the scale in use. The rule is written before the run. Recompute the scale every fixed number of steps from the window's maximum, and keep the previous scale if the new one differs by less than a factor of two. Promote a tensor to a wider format when its updates sit below its step for a whole window.
The claim is then stated in two parts. The run did not overflow: every tensor stayed inside its format's range, which the scale rule enforces by construction. The run matched a wide reference to a stated tolerance. On the loss and on a named metric, over the same number of steps and the same data order, the narrow run stays within a tolerance fixed in advance. A relative difference of one per cent at the end of training is one such tolerance. The second claim is the one the procedure controls. It is not the same as reaching the same quality on held-out data, which the answer does not promise because the narrow arithmetic changes the trajectory.
The failure is named with its mechanism. A per-tensor scale recomputed from the current batch makes the arithmetic depend on the batch order, so two runs that shuffle differently are not the same computation, and a spike in one batch changes every tensor's next step. A procedure that watches only the maximum misses swamping, because a weight of 1.0 with an update of 0.001 changes nothing while its magnitude is comfortably in range. A global scale chosen for the embedding cannot serve a layer whose gradients are a thousand times smaller. A scale that is not a power of two introduces a rounding of its own. A scale that is a power of two leaves up to a factor of two of the range unused, which matters at 8 bits.
The cost is stated. A master copy in 32 bits means four bytes for every weight alongside the two-byte copy, which is the traffic the narrow format was meant to cut. A windowed maximum needs a pass over the tensor or a running statistic, and a scale that answers a spike one window late lets the spike overflow. The answer says which of these it pays and why the purchase is worth it for its purpose. That combination — a measured quantity, a written rule, a claim with a tolerance, a named failure and a stated cost — is what separates level 3 from a preference for 16 bits.