The training loop

A training run prints the same number after every pass, and the number is 0.6931 = ln 2. That value is the log loss — the binary cross-entropy — of a network that answers 0.5 to every example when the two classes are equally frequent; it is the loss of pure guessing under that loss and that balance. Under the squared error of §19.5, the same constant answer would print one eighth. A constant answer cannot be confidently right and confidently wrong at the same time, and a loop that prints this value forever is running correctly and learning nothing.

What a network learns is a value for each of its parameters, and the loop is the procedure that adjusts them. The adjustment is tiny and repeated. Measure how the loss responds to each parameter, then move every parameter a little in the direction that lowers the loss [S:67:39]. The size of the move is the learning rate, and the direction comes from the gradient computed by backpropagation. !The same loss under two step sizes. At 0.1 the parameter walks 1, 1.4, 1.72, 1.976, and the tangent at each stop shows the step as the slope times 0.1, so the loss falls from 4 to 2.56 and then to 1.6384. At 1 the walk jumps 1, 5, 1, because the step overshoots the minimum by as far as it started, so a larger rate does not buy a faster arrival. Status: re-simulated.

$$\theta \to \theta -\eta \nabla J(\theta )$$ !The order of one pass. Parameters are set from a recorded seed, the example order is shuffled, a batch is drawn, the stored gradients are cleared, the forward pass keeps its intermediate values, the backward pass fills the gradients and the update is applied. The measurement sits outside the loop, after the parameters are frozen, because a loss read halfway through an update describes neither the weights before the step nor the weights after it. Status: schematic.

That line says the new parameters are the old ones minus the learning rate times the gradient, so each step walks downhill on the loss surface.

Worked example. One parameter, one loss, three steps. Let the parameter be w, let the loss be (w - 3)^2, and start at w = 1 with a step size of 0.1. The slope at w = 1 is -4, so the first update moves w to 1.4. The slope there is -3.2, so the second value is 1.72, and one more step gives 1.976.

The loss falls from 4 to 2.56 and then to 1.6384, and the parameter is crawling toward the value 3, where the loss would be zero.

The loop around that arithmetic has more steps than the arithmetic suggests, and the order is part of the specification. Set the parameters from a random generator with a recorded seed. Shuffle the example order at the start of each pass, because a fixed order teaches the network whatever arrives first [S:47:45]. Draw a batch, then clear the stored gradients. Run the forward pass and keep the intermediate values, compute the average loss on the batch, run the backward pass, and apply the update.

Only after the pass ends, with the parameters frozen, measure the training loss and the held-out loss. That final measurement describes the model you actually have, and it is the number the next chapter will ask you to read.

Two of those steps exist for reasons that a derivation never shows. Clearing the gradients is a requirement of the software, not of the mathematics: the chain rule of §19.5 carries no state between calls, so only the framework's programming interface decides what a backward call leaves behind. Read the clearing step in your library's documentation before you change it. Measuring after the pass is a requirement of interpretation, because a number read halfway through an update describes neither the parameters before the step nor the parameters after it.

A pass and an iteration are also different objects, and the difference matters when you read a training curve. One pass visits every example once; one iteration performs one update, which may cover a handful of examples or a thousand. A curve labelled in passes and a curve labelled in iterations can describe the same run while looking nothing alike, so a reported curve without its unit is not evidence of anything.

The batch size introduces a further trade. A small batch gives a noisy estimate of the gradient, which is cheap and occasionally helpful, since the noise can push the parameters out of a flat region. A large batch gives a stable estimate at a higher cost per step, and the average over the batch is what makes the step size comparable across batch sizes. The learning rate then decides the character of the walk, and a rate that is too large produces violent oscillations in the loss curve [S:66:267].

The word learn deserves one caution. Nothing in the loop stores examples, and nothing in it reads a rule off the data in the way a person would. The parameters settle into values that make the loss small on the batches they see, and the same values may behave quite differently on examples the loop never saw. That gap is why the measurement after the pass matters more than the loss during it, and why a falling training curve is evidence about the optimizer rather than evidence about the world.

The formal statement. Let $J$ be the loss and $\theta$ the parameter vector. Stochastic gradient descent draws a batch, forms an estimate $\hat{g}$ of the gradient, and updates $${\theta }_{t+1}={\theta }_t-{\eta }_t{\hat{g}}_t.$$ The estimate is unbiased when the batch is drawn at random. The average of the per-example gradients has the gradient of the average loss as its expectation [S:66:266]. Convergence of the stochastic recursion needs its own theorem. The Robbins–Monro conditions are: step sizes that go to zero, step sizes whose squares sum to a finite number, and step sizes whose sum diverges [S:58:116]. The noise must have zero mean and finite variance. Under those conditions the iterates converge to a root with probability one.

The derivation. The Core's quadratic gives the whole picture in one line. Let the loss be $(w-3{)}^2$, so the slope is $2(w-3)$. One step sends $w$ to $w-2\eta (w-3)$. Write the error as $e=w-3$. The update becomes $e\mapsto (1-2\eta )e$, so the error shrinks exactly when $|1-2\eta |<1$, which is $0<\eta <1$. At $\eta =1$ the factor is $-1$ and the error flips sign without shrinking. The Core's stop-and-try at six tenths sits inside the stable range, where the error shrinks and crosses the minimum on every step. For a general smooth loss with gradient Lipschitz constant $L$, the same algebra gives stability when $\eta <2/L$.

The edge case. A fixed step size and a decaying step size answer different questions. With a fixed step the noise never vanishes, so the iterates settle into a neighbourhood of the minimum whose width scales with the step and the noise, not onto the minimum itself. With a decaying step that satisfies the Robbins–Monro conditions the noise is averaged away and the iterates converge. A run with a constant learning rate is therefore not a failed run; it is a run answering a different question.

What the Core hid. The number $0.6931$ is the natural logarithm of two, which is the loss of a network that answers one half to every example under a log loss. The loop prints it forever because the parameters sit where the gradient is tiny or where the two classes cancel, not because the code is wrong. The Core's rule about measuring after the pass is an interpretation rule, and it prevents a reader from blaming the optimizer for a number that describes a half-finished step.

Where the analogy breaks. Walking downhill assumes a fixed landscape and a walker who can see the slope. The landscape here is the average over batches, the walker sees one batch, and a step that lowers the batch loss can raise the average. Downhill is a local and noisy statement, and the guarantee is about the average motion rather than about any single step.

Original sources. Stochastic approximation is Robbins and Monro's, and Bishop gives the conditions on the step sizes with the almost-sure convergence that follows [S:58:116]. Goodfellow and coauthors state the mini-batch estimator and the convergence rates [S:66:266] [S:66:267]. The perceptron is the same recursion applied to a convex and non-differentiable loss [S:75:208].

An open question. The optimization view predicts the training loss and does not predict the test loss. Bottou and Bousquet argue that converging faster than a stated rate may correspond to overfitting, so a faster optimizer is not automatically a better learner [S:66:268]. The unsettled question is which measurable property of the trained parameters predicts the gap. Evidence that would settle it: a quantity computed from the trained model alone that predicts test error across architectures at fixed data, with the prediction recorded before the test set is read.

A frontier result. For convex problems the excess error after k iterations is on the order of one over the square root of k, and in the strongly convex case one over k [S:66:267]. Those rates are the yardstick the field applies to every new optimizer. They are also why a run can look slow and still sit close to the best error reachable at that step size. A bound of this kind describes an average over runs, not the curve you see from one seed.

What to read next. Read Goodfellow 8.3 for the estimator, the rates, and the argument about faster convergence [S:66:266] [S:66:267] [S:66:268]. Read Bishop's treatment of the Robbins–Monro procedure for the theorem in its original setting, which is root finding under noise [S:58:116]. Read Mohri's chapter on linear classification to see the perceptron as one instance of the recursion [S:75:208]. Then read the derivation in Zhao, where the hypotheses on the step sizes are stated and used [S:82:119].