What do momentum and Adam add?

Give a weight the same gradient, 1, at every step and set the momentum coefficient, the fraction of the previous velocity that carries over, to 0.9: the velocity, the running update that momentum accumulates, goes 1, then 1.9, then 2.71, then 3.439, and it approaches 10 [S:63:317]. Plain descent, which uses the gradient alone, takes steps of size 1 forever, while momentum ends at ten times that, without any change to the gradient itself.

Momentum replaces the current gradient with a running average of past gradients, so a direction that repeats accumulates speed [S:63:317]. Adam keeps a second average, of the squared gradient, and divides each step by the square root of that average [S:171:1]. Both are add-ons to stochastic gradient descent, the update that moves each weight by the learning rate times its gradient. !Under a steady gradient the velocity climbs toward 1/(1-beta), which is 10 at 0.9 and 100 at 0.99; the slower curve is still far from its limit after twenty steps. Reversing the gradient every step leaves the velocity oscillating near zero, so momentum speeds up a direction only while the direction holds. Status: re-simulated.

The section question asks what they add: momentum adds memory of direction, which helps in a long narrow valley where the gradient points across the valley more than along it [S:63:317]. Adam adds memory plus a per-weight scale, so the step size stops depending on the size of the gradient [S:171:1]. Neither changes the direction the loss points in, and both change how far and how smoothly you travel. !Plain descent moves the weight with gradient 10 a hundred times as far as the weight with gradient 0.1. Adam divides each step by a running measure of that weight's gradient size, so both take a step of roughly the learning rate. Status: schematic.

The per-weight scale descends from two earlier methods. AdaGrad divides by the square root of the sum of all squared gradients seen so far, so weights that have already moved a lot slow down. RMSProp replaces that growing sum with a decaying average, which lets the rate recover after a steep patch [S:66:280]. Adam is the combination of that decaying second average with a decaying first average, which is why it behaves like momentum and RMSProp at once [S:66:281].

Adam's averages start at zero, which makes them too small at the first steps, and a bias correction divides each average by one minus its decay raised to the step count [S:171:2]. With a gradient of 2 and a first-moment decay of 0.9, the raw average after one step is 0.2, and dividing by 0.1 restores 2, the value the gradient actually had.

The second average is the second moment, the running mean of the squared gradient [S:66:281]. Its square root is the typical size of the gradient, so dividing by it turns the update into a step of roughly the learning rate, whatever the gradient's scale. Two weights whose gradients differ by a factor of 100 then take steps of comparable size under Adam, while plain descent moves the first one 100 times faster. That is the whole benefit: the learning rate you set becomes the step you get, rather than a number that must be rescaled for every layer. Adam also handles gradients that are mostly zero, since a weight whose gradient is rarely nonzero still receives a full-sized step when one arrives, and the original paper lists sparse gradients among its design targets [S:171:1].

Worked example. Hold the gradient at 1 for five steps and apply the momentum rule with a coefficient of 0.9, so that each velocity is 0.9 of the old one plus the new gradient.

Each step keeps nine tenths of the old velocity and adds the gradient, so the sum 1 + 0.9 + 0.81 + ... equals 1 divided by 1 minus 0.9, which is 10 [S:63:317]. That limit is why the coefficient is called a decay: older gradients count for less and the total memory is finite, while a coefficient of 0.99 would give a limit of 100 and a memory of the last hundred steps or so.

Adam stores two numbers for every weight, the first and second moments, so its optimiser state is twice the size of plain momentum's and two numbers per weight more than plain stochastic gradient descent, which stores none [S:66:282]. The combination of momentum with rescaling has no clear theoretical justification [S:66:281], and the learning rate still needs tuning for the problem at hand [S:66:281]. Momentum alone is cheaper, and on some problems it matches Adam.

The formal statement. Momentum keeps a velocity and updates it with a decayed sum of past gradients:

with the decay $\beta$ between zero and one, the learning rate $\eta$, and $g_i$ the gradient at step $i$ [S:63:317]. Adam keeps two averages of the gradient, the first and the second moment, corrects both for their zero start, and divides the first by the square root of the second [S:171:2]. The hypotheses are a fixed decay per moment and a step size small enough that the linearisation below is meaningful.

The derivation. Unroll the momentum recursion and the velocity becomes a weighted sum of every past gradient, with weights that fall by the decay each step. A constant gradient turns that sum into a geometric series with limit the gradient divided by one minus the decay. That limit is the factor of ten L1 quotes for a decay of 0.9 [S:63:317]. For Adam, take expectations of the second-moment recursion and assume the gradient distribution is stationary. The expectation is the true second moment times one minus the decay raised to the step count. Dividing by that factor removes the bias the zero start introduces [S:171:3]. That is the whole content of the correction, and stationarity is the load-bearing assumption.

The edge case. Hold the gradient at a constant value. The corrected first moment then equals the gradient, and the corrected second moment equals its square. The update is the learning rate in the direction of the gradient's sign, whatever the size of the gradient. L1's "the step is roughly the learning rate" is exact here, and the same fact is the failure mode. The bound the paper proves has two cases, with $\epsilon =0$: the absolute step is at most $\alpha (1-{\beta }_1)/\sqrt{1-{\beta }_2}$ when $(1-{\beta }_1)>\sqrt{1-{\beta }_2}$, and at most $\alpha$ otherwise [S:171:2]. With the paper's defaults, ${\beta }_1=0.9$ and ${\beta }_2=0.999$, the first case holds, so the bound is $0.1/\sqrt{0.001}\approx 3.16\alpha$: a weight whose gradient is almost always zero can take a step about three times the learning rate. The full-size-step failure needs the gradient to be small but consistent, since only then does the corrected ratio tend to a fixed sign whatever the magnitude. A small noisy gradient makes the ratio fall, which is what the paper reports: "the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps in parameter space" [S:171:3].

The second edge case is the small constant in the denominator. Once the square root of the second moment falls to the size of that constant, the update becomes linear in the gradient again, and the scale invariance is gone.

What L1 hid, and why that was safe there. L1 said the combination of momentum and rescaling has no clear theoretical justification [S:66:281]. The stronger statement is that the theorem the original paper proved does not hold for the algorithm as published: Reddi et al. (2018) gave a convex problem on which Adam diverges, and their AMSGrad modification restores convergence [S:74:107]. A proof is not refuted by an example; a theorem is, and this one was. L1's examples use a constant or an alternating gradient, where the correction is exact and the counterexample cannot appear.

Where the analogy breaks. "Memory of direction" suggests that momentum remembers where the loss is low. It remembers the gradient, not the surface, so it can carry a weight past a minimum and back.

Original sources. Momentum is the heavy-ball method, stated with its geometric-series limit in Murphy section 8.2.4 [S:63:317]. AdaGrad and RMSProp are the two ancestors of the second moment: AdaGrad divides by the square root of the accumulated squared gradient, and RMSProp replaces the sum with a decaying average [S:66:280]. Adam combines the two averages and adds the bias correction [S:171:1].

An open question. Adam's convergence is not settled for the problems people use it on. The original paper proved a regret bound for convex objectives, and a theorem cannot survive a counterexample: Reddi et al. (2018) exhibit convex problems on which adaptive methods with a non-decreasing second-moment average fail to converge, and they propose AMSGrad, the first repair [S:63:331]. Padam (Chen et al., 2018; Zou et al., 2018), Yogi (Zaheer et al., 2018) and later ADOPT (Tan et al., 2024) follow; Yogi's contribution is a convergence proof for the nonconvex case, not another convex counterexample [S:74:107]. What would settle the practical question is a suite of small nonconvex problems where vanilla Adam diverges from every reasonable initialisation, paired with a proof of convergence under assumptions those problems satisfy. The practical question is narrower: which of the two averages carries the benefit on the models people train, and at what scale the answer changes.

A frontier result. Decoupled weight decay separates the regularisation from the adaptive step. In Adam the L2 penalty is scaled by the same second-moment factor as the gradient, so the decay is weaker for weights with large gradients. AdamW applies the decay directly to the weights [S:181:1]. With adaptive gradients, L2 regularization penalises weights with large historic parameter or gradient amplitudes less than weight decay does, which is the mechanism behind the difference [S:181:1].

What to read next. Read section 8.4.6 of [S:63:330] for the recursion and the bias correction. Read section 8.5 of [S:66:281] for the comparison with RMSProp and the remark that the combination lacks a theory. Read the Adam paper [S:171:1] for the step bound. Then read the decoupled-decay paper [S:181:1] for the modern default. Together the four readings separate the algorithm from the folklore around it.