Debugging a training run

Six steps of a run report losses of 2.00, 1.60, 1.90, 1.50, 1.70 and 1.40. Watch single steps and step 3 sits above step 2, and step 5 above step 4. Average three steps and the smoothed run still turns up at step 3 and again at step 5: smoothing shrinks each reversal, it does not remove it.

A loss curve is the plot of the training loss against the step number, usually with the validation loss drawn on the same axes. Its overall shape answers the section's question, which loss curve means which bug, because each common failure leaves a different signature in the curve and in the gap between the two lines. !Each panel is a sketch of a shape, not a recorded run: flat and high, a widening gap between training and validation, a rise that ends in NaN, and a single spike that recovers. The shape narrows the cause to a short list, and a controlled rerun that changes one thing names which. Status: synthetic.

A training loss that stays high and flat points to underfitting, a model too small or too weakly trained to fit the data [S:69:155]. It can also indicate a bug such as a mislabelled target, a dead layer, one whose units all output zero, or a learning rate so small that progress stops [S:66:389]. The two possibilities resemble each other on the curve, which is why the initial diagnostic is whether the model can fit a handful of examples at all.

A training loss that falls while the validation loss climbs is overfitting [S:69:155], and the gap is the signal rather than the training number alone, because a low training loss never certifies a low validation loss. A validation curve that falls and then turns upward marks the point where the model stopped learning the pattern and started memorising the sample [S:69:155].

A loss that climbs instead of falling usually indicates that the learning rate is too large [S:66:389]. A loss that becomes infinite or not a number, written NaN, is a numerical failure: the computation has left the range the dtype can represent. It is usually the endpoint of an exploding gradient, but one operation, such as a logarithm of zero in a cross-entropy or a division by a zero variance, produces it with no divergence at all.

Divergence is the weaker statement that the loss is rising and not returning [S:645:1]. A non-finite value can be handled by clipping the step or skipping the batch, which the source prescribes for that case even when the gradient is already Inf or NaN [S:66:375]. A single narrow jump that recovers is a loss spike. The proxies paper sets spikes aside from the slow divergence it studies and returns to them in its section 4, so the loss curve alone does not name the cause [S:645:2]. The nearest thing to a cause it reports is the optimiser's own state: the adaptive second-moment estimate can go stale, a quantity the loss does not plot, and reducing its decay resolves the spike [S:645:10-11].

The curve also carries the batch size: a batch of 32 examples gives a gradient whose standard deviation is 4 times larger than the gradient from a batch of 512. The standard deviation of an average falls with the square root of the number of examples averaged. The same run plotted at two batch sizes therefore looks like two different runs, so verify that two curves share a batch size, a schedule and a number of examples seen before comparing them.

Worked example. Smooth the six losses by averaging each window of three, writing the window at the step where it ends.

The three-point trailing average is 2.00, 1.80, 1.83, 1.67, 1.70, 1.53. It rises at steps 3 and 5, the same two steps as the raw curve, so smoothing does not turn the run into continuous improvement: it cuts the size of each reversal and lowers the noise of the estimate. The smoothed curve falls only at steps 4 and 6.

The window is a choice. A trailing window of two gives a monotone series here, so two points already remove a one-step reversal. A longer window buys a smaller variance, since the noise of the average falls as one over the square root of the window, at the cost of a lag of about half the window. With a window of k steps the smoothed series turns up whenever the newest point is above the point that leaves the window.

Before trusting any curve, log the gradient norm, the length of the gradient vector, for the first and last layers. A norm that falls by orders of magnitude across the depth is the vanishing gradient of the first section. An instability often appears in these norms before it is visible in the loss, so the norms are the early warning and the loss is the report [S:645:2].

The shape narrows the list of causes, and it does not name one, since two different bugs can produce the same curve and a curve on a small subset can look healthy while the full run fails [S:645:1]. The check that settles it is a controlled rerun with one change at a time, which costs compute and is the real price of an honest baseline.

The formal statement. Let the reported loss at step $i$ be the value the optimiser produces plus the noise of one batch. A trailing moving average with window $k$ replaces each point by the mean of that point and the $k-1$ before it. If the noise terms are independent with variance ${\sigma }^2$, then

Read it as: the standard deviation of the noise is $\sigma /\sqrt{k}$, so it falls as one over the square root of the window, while the mean of the window lags the true curve by about $(k-1)/2$ steps. Both facts follow from the weights the window assigns.

The derivation. The variance of an average of independent terms is the average of their variances, because every covariance is zero. The lag follows from the same weights: the window spreads its mass over the last $k$ steps, and its centre of mass sits $(k-1)/2$ steps behind the current step. The step I omit is the independence. Consecutive batches are drawn from a shuffled set without replacement, so they are slightly negatively correlated and the true variance is a little smaller than the formula.

The edge case. The window trades noise against lag: a longer window lowers the noise and pushes the curve further behind, and the two requirements can conflict. Early in training the loss falls fast and the noise is large. Late in training the loss is flat and the noise is small. A window chosen for the first phase lags in the second, and a window chosen for the second is noisy in the first. Monotonicity is not a property of smoothing at any finite window: the six-point table in L1 has a window of three and its smoothed line still turns up at step three and again at step five.

A second edge case is the single spike. One unusual batch raises the average for three steps, and one non-finite value turns every later weight into a non-finite number. The curve then shows a vertical jump, and lowering the learning rate does not repair a value that is already not a number. Gradient clipping bounds the step before it is taken [S:66:375], and a spike that recovers on its own sends you to the second-moment state rather than to the learning rate [S:645:10-11].

What L1 hid, and why that was safe there. L1 read the curve as a report and the smoothed curve as the same report at a coarser scale. The refinement is that smoothing is a filter with a known variance and a known lag, and both depend on a window you chose. In the worked example the noise was small and the window was fixed, so the choice did not matter there.

Where the analogy breaks. A report describes an event that has happened. A loss curve is one sample path from a random process, and a second run of the same model gives a different path.

Original sources. The diagnostic reading of training and validation error is chapter 11 of Goodfellow, Bengio and Courville [S:66:389]. Their account also gives the quadratic case: a learning rate at least twice the optimal value increases the training error instead of decreasing it [S:66:389]. The modern study of instability at scale is the small-scale-proxies paper [S:645:1].

An open question. Nobody has settled whether the instabilities of large training runs can be predicted from small ones. The paper reproduces two known instabilities in small transformers by raising the learning rate, and it predicts one of them from the scaling of activation and gradient norms before the loss moves [S:645:2]. What is missing is a rule that says which instabilities transfer and which do not. The evidence that would settle it is a sweep over model size at fixed hyperparameters, with the instability threshold measured at every size and the norm trajectory recorded before the threshold. If the threshold follows a power law in size, prediction is possible. If it jumps, the small-scale proxy is a coincidence. A negative result would matter as much as a positive one, because it would say which failures exist only at scale.

A frontier result. Learning-rate sensitivity summarises a whole sweep as one number: the expected deviation from the best loss when the rate varies over three orders of magnitude [S:645:2]. That turns "this model is unstable" into a quantity you can plot against scale. The paper uses it to show that scaling depth raises sensitivity faster than scaling width [S:645:6].

What to read next. Read chapter 11 of [S:66:388] for the capacity reading of the two curves, and the page that follows for the learning-rate argument [S:66:389]. Read the proxies paper [S:645:1] for the taxonomy of instabilities. Then read the identity-mapping paper [S:179:1], because the gradient norms that warn you are the norms a skip connection changes.