Forward mode

A single number comes out, and two parameters go in. Take f(θ1, θ2) = θ1θ2 + θ1 at θ1 = 2, θ2 = 5. The value is 12, and a training step needs both slopes: how f moves when θ1 moves, and how it moves when θ2 moves. Forward mode computes one of those slopes per run, by carrying a rate beside every value as the program runs.

The rate is called the tangent, and the value it belongs to is the primal value. A dot over a letter marks the tangent, so the pair for the first parameter is θ1 = 2 and its tangent is 1. At the inputs you choose the direction yourself: set the tangent to 1 for the parameter you are probing and to 0 for the rest. Everything downstream follows from the chain rule. !Every value carries a rate beside it. In the left panel the seed is (1, 0), so ${\theta }_1$ enters at rate 1 and ${\theta }_2$ at rate 0; the product passes $1\cdot 5+2\cdot 0=5$, and the sum adds the direct edge to return 6, the slope in ${\theta }_1$. In the right panel the seed is (0, 1) and the same program returns 2, the slope in ${\theta }_2$. Two parameters, two passes, and each pass is one column of the Jacobian. Status: schematic.

One pass returns one Jacobian-vector product, which is the derivative of the output along the chosen direction [S:67:265]. It is a linear combination of the columns of the Jacobian, with the components of the direction as coefficients. It is the j-th column exactly when the direction is the j-th unit vector — the seeding Bishop uses to compute the Jacobian one column at a time [S:67:266]. For any other direction it is not a column of the Jacobian. A single complete evaluation of the program carries a second sequence of numbers beside the primal values, and that second sequence costs a fixed amount of arithmetic per operation.

The chain rule is the only machinery. Each operation carries a local rule: a sum passes both tangents through unchanged, and a product multiplies each tangent by the other factor. The sweep applies those local rules in execution order, so every tangent exists before a later operation reads it. No step is chosen, so no truncation error enters; the arithmetic still rounds, and no symbolic derivative expression is ever constructed.

The same pattern covers every primitive. For a sum the tangent is the sum of the tangents, and for a product the rule below applies. For an exponential the tangent is the input tangent multiplied by the output value, because the derivative of an exponential is the exponential itself. A table of such local rules is everything an implementation requires.

A direction is a list of instantaneous rates, one for each input coordinate. The two standard directions (1, 0) and (0, 1) are the two coordinate axes, and linearity builds every other direction from those two. The Jacobian-vector product is not the gradient; it is one number per output, and for a scalar loss it is a single number that depends on the direction you chose.

Worked example. Start with θ1 = 2 and tangent 1, then θ2 = 5 and tangent 0. The product p = θ1θ2 has value 10, so its tangent is 1·5 + 2·0 = 5. The sum f = p + θ1 has value 12, so its tangent is 5 + 1 = 6.

That 6 is the slope with respect to θ1, and the formula agrees: differentiate θ1θ2 + θ1 with respect to θ1 and you get θ2 + 1 = 6. A second pass starts from tangents 0 and 1. The product's tangent becomes 2, the sum's tangent stays 2, and that is the slope with respect to θ2.

The rule for a product is worth writing once. For p = θ1θ2 the rule reads as follows.

That line says: the rate of a product is the rate of the first factor times the value of the second, plus the value of the first times the rate of the second.

Two passes produced two slopes, and the number of passes is the entire story. The full Jacobian needs one pass for each input [S:67:266], so a function with a thousand parameters needs a thousand runs. Forward mode is efficient when the inputs are few and the outputs are many, which is the opposite of a loss function. For a scalar loss with a million parameters, the forward sweep would need a million traversals of the same program.

The memory requirement is comparatively modest. The primal and tangent values are computed together, so a pair can be discarded as soon as its last reader is done [S:67:267]. Memory does not grow with the depth of the program, which is the property that makes the sweep attractive for long chains of operations. It grows with the width of a layer, because a wider layer carries a wider second sequence, and it stays a small multiple of what the program needs anyway.

The arithmetic expense is also modest for each evaluation. Bishop reports that one pass costs no more than six times a function evaluation, and usually about two or three [S:67:267]. The constant factor is not the difficulty for forward mode. The number of passes is.

Reading the slopes off a formula works for the two-parameter example and becomes unmanageable for a network. The loss of a deep model is a composition of thousands of operations, and its symbolic derivative is a far larger composition. The tangent sweep never forms that object at all, so its cost follows the program rather than the formula.

Limitation: The price is set by the number of inputs, and a modern model has millions. Forward mode also answers only the direction you seeded; it never hands you the whole gradient in one run. Section 18.3 turns the sweep around and fixes both problems.

The formal statement. Let $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$ be differentiable at $x$, with Jacobian $J(x)$ of shape $m\times n$. Forward mode computes, for one chosen direction $v\in {\mathbb{R}}^n$, the Jacobian-vector product $J(x)v$ [S:67:265].

Give each value a partner. Write a pair $(a,\dot{a})$ and let a primitive $\phi$ act by $(a,\dot{a})\mapsto (\phi (a),{\phi }^{\prime }(a)\dot{a})$. Sums and products act by $(a,\dot{a})+(b,\dot{b})=(a+b,\dot{a}+\dot{b})$ and $(a,\dot{a})(b,\dot{b})=(ab,\dot{a}b+a\dot{b})$. The pair is a jet of order one: the value and its rate along the seeded direction. The hypothesis is that every primitive has a derivative on the values the run reaches.

The derivation. Induct over the nodes of the graph, not the positions of a chain. For a node $i$ with parents $\mathrm{pa}(i)$, the carried tangent is

which is Bishop's equation (8.57) [S:67:264]. The induction hypothesis is that ${\dot{v}}_i$ is the directional derivative of the value at node $i$ with respect to the seeded direction. The chain composite $f_t\circ \cdots \circ f_1$ is the special case in which every node has exactly one parent. With the sum over parents the induction goes through unchanged, and the base case stays $\dot{v}=v$.

At the output node the carried tangent is $J(x)v$. No step was chosen, so no truncation error enters; the arithmetic still rounds. The induction is the proof; the only omitted step is the differentiability hypothesis.

The edge case. A zero directional derivative does not mean a flat function. Take $f({\theta }_1,{\theta }_2)={\theta }_1-{\theta }_2$ and seed the direction $(1,1)$. One pass returns zero, while neither partial derivative is zero. The core's sentence "one pass returns the derivative along the chosen direction" is true, and a reader can still hear "the derivative" as "the gradient". The two agree only when $v$ is a coordinate axis.

The same case shows the scaling. Seed $(2,0)$ and the answer doubles while the primal value does not move. The tangent is a rate per unit of $v$, not a property of the function.

What the core hid. The core said the full Jacobian needs one pass per input, and that is the right count for a gradient. It is not the whole cost story. One pass returns $m$ numbers, one per output, so the price is $1/m$ of a pass per returned number. The full Jacobian costs $n$ passes and returns $n\cdot m$ entries, which is again $1/m$ of a pass per entry: forward mode is cheap per number whenever $m>1$, and a vector output is cheaper per number than the count of passes suggests. The scalar loss $m=1$ is the worst case, where each number costs a whole pass; it is also the best case for reverse mode.

Where the analogy breaks. "Carrying a rate beside every value" sounds like a velocity attached to a moving particle. It breaks under a change of direction. The pair is linear in $v$, so doubling the seed doubles the tangent and leaves the primal trajectory untouched. A velocity would change where the particle goes; the tangent does not change the value it is attached to.

Original sources. Blondel and Roulet give forward mode as Algorithm 8.1: carry the linearisation point and the perturbation through the chain, and update the perturbation by the local Jacobian at every step [S:72:200]. Bishop builds the same sweep as an evaluation trace in which every primal variable carries a tangent, and writes the tangent recursion as a sum over parents [S:67:263]. Both are the chain rule applied in execution order, and neither forms a derivative expression.

An open question. Whether a randomised forward-mode estimator can replace reverse mode when memory is the binding constraint. Draw a Gaussian direction $Z$ and average $⟨\nabla \ell (\theta ),Z⟩Z$ over samples. The result is an unbiased estimate of the gradient that uses JVPs alone [S:72:229]. Its variance is high, and how much variance reduction is needed before it beats a checkpointed reverse pass at equal memory is not settled.

What would settle it: training curves at matched memory and matched wall-clock, with the estimator's variance measured directly rather than inferred from the loss. A comparison that gives the randomised estimator less memory and calls the contest fair has not tested the claim.

A frontier result. Hessian-vector products by a hybrid of the two modes. Seed a tangent, run the forward sweep to obtain the directional derivative, then differentiate that scalar with reverse mode to obtain $Hv$. Bishop reports $O(W)$ cost for $W$ parameters, although the Hessian has $W\times W$ entries [S:67:267]. The method depends on forward mode being nestable inside reverse mode, which is the composition property the cost of forward mode alone does not show.

A second bound belongs here. One pass costs no more than six function evaluations and usually two or three [S:67:267]. That constant is the reason a forward sweep is affordable per pass, and the pass count is the reason it is not affordable per gradient.

What to read next. Blondel and Roulet §8.1.1 for the algorithm and §8.7 for the randomised estimator and its variance [S:72:229]. Bishop §8.2.1 for the evaluation-trace presentation and the two-output example that shows one pass returning a whole column [S:67:265]. Then read §8.3.1 of Blondel and Roulet for the circuit version, where fan-out first appears.