Open: Change, Slopes and Approximation

There is no single correct answer, so this is a model of the shape a complete answer takes, not a key. A level-3 answer starts from what the chapter already proves. For a quadratic with curvature $L$ along a direction, the error after one step is multiplied by $1-\alpha L$, so the step is safe exactly when $0<\alpha L<2$, and the error is removed in one step at $\alpha =1/L$. That gives a rule with a quantity in it: estimate the largest curvature $L$ you expect along the directions you travel, and set $\alpha$ below $2/L$, with a margin that puts it nearer $1/L$ because the estimate can be wrong. The rule optimises a guarantee of decrease per step, and its cost is the work of estimating $L$, which needs either a second derivative, a measured change in the gradient, or a search that evaluates the loss several times per step.

The answer then says what the guarantee does not cover. A loss with two curvatures, one large and one small, permits a step that is safe for the large one and slow for the small one; the run zigzags across the narrow direction while creeping along the valley. A step chosen for the average curvature can therefore satisfy the local bound and still take many steps. A search along the gradient adapts to the local curvature and costs extra evaluations, and it can accept a step that lowers the sampled loss while raising the full loss when the gradient comes from a batch. A flat region makes the gradient small in every direction, so a rule phrased only in terms of the gradient's length can shrink the step to nothing.

Finally the answer names evidence. A loss curve that falls and then flattens, a gradient norm that stays large while the loss moves little, a comparison of the step's predicted fall with the observed fall, and a check on a batch held out of the step all bear on the choice. None of them settles it for every loss, which is why the question remains open: the practitioner chooses among procedures that trade a guarantee, a cost and a failure mode, and states which trade the problem at hand can afford. Rubric: - 0 — Restatement. The answer says that the step size matters and that a large step can diverge, but names no rule, no quantity and no evidence. Nothing is decided. - 1 — A number or a schedule with no quantity. The answer gives a value such as $0.01$, or says to halve the step every so often, without saying what the value is measured against, what it optimises, or what would show that it fails. Curvature and the scale of the gradient are not mentioned, or are mentioned without consequence. - 2 — A rule tied to the loss. The answer gives a rule with a quantity in it, such as $\alpha \le 1/L$ with $L$ an estimate of curvature, or a search that accepts a step only when the loss falls by a fixed fraction of the predicted fall, or a step of fixed length along the normalised gradient. It says what the rule optimises, such as guaranteed decrease or speed, and what it costs in extra evaluations or tuning. It names one failure and one measurement, such as a loss trace, a gradient norm, or a check on held-out data. - 3 — Rule, guarantee and failure defended. The answer does everything at level 2 and also derives the safe range for a quadratic, $0<\alpha L<2$, and uses it to explain why the rule is stated in terms of curvature rather than the raw step. It separates the local guarantee from behaviour over a whole run, and it names a concrete failure with its mechanism: a step above $2/L$ diverges, one safe for the steepest direction zigzags in an elongated valley, a sampled gradient can point away from the full gradient, or a flat region makes the direction unreliable. It states the cost of its own procedure and the evidence that would make it change the choice, and it admits that no rule dominates for every loss.