Practise: Getting Gradients From a Computer

At $x=3.1$ the function is $3.1^2=9.61$, and at $x=2.9$ it is $2.9^2=8.41$. The one-sided quotient is $(9.61-9)/0.1=0.61/0.1=6.1$. The centred quotient is $(9.61-8.41)/0.2=1.2/0.2=6.0$. The exact derivative is $2x=6$ at $x=3$, so the one-sided estimate is high by $6.1-6=0.1$. The centred estimate is exact here for a reason worth naming: for a quadratic, the curvature on the right cancels the curvature on the left, so the centred quotient returns the slope with no truncation error. The same quotient on $x^3$ at $x=2$ misses by ${\Delta }^2$, because a cubic does not have that symmetry. Section 18.1 uses the cubic case to show both errors; this quadratic case separates the one-sided bias from the centring.

Forward mode costs one pass for each input direction, and the model has 250 parameters, so it needs 250 passes. Reverse mode costs one pass for each output, and the scalar loss is one output, so it needs one pass. The full gradient has 250 numbers either way; the two modes reach it from opposite sides of the same Jacobian. Forward mode is the better choice when the inputs are few and the outputs are many. A loss function is the opposite case: one output and a large input count, which is why reverse mode is the mode training uses. Section 18.2 counts the forward passes, and section 18.3 counts the reverse pass.

The input pair is ${\theta }_1=3$ with tangent 1, and ${\theta }_2=2$ with tangent 0. The square $s={\theta }_1^2$ has value $3^2=9$. Its local rule is $2{\theta }_1{\dot{\theta }}_1=2\cdot 3\cdot 1=6$, so the pair for $s$ is value 9, tangent 6. The product $f=s{\theta }_2$ has value $9\cdot 2=18$. Its tangent is $\dot{s}{\theta }_2+s{\dot{\theta }}_2=6\cdot 2+9\cdot 0=12$. The formula agrees: $\partial f/\partial {\theta }_1=2{\theta }_1{\theta }_2=2\cdot 3\cdot 2=12$. This one pass returned the derivative along the direction $(1,0)$, which is the slope in ${\theta }_1$. A second pass with direction $(0,1)$ would give the slope in ${\theta }_2$, which is ${\theta }_1^2=9$. The two passes together are the two columns of the Jacobian.

The product $p={\theta }_1{\theta }_2$ has value $0\cdot 1=0$. Its tangent is ${\dot{\theta }}_1{\theta }_2+{\theta }_1{\dot{\theta }}_2=1\cdot 1+0\cdot 0=1$. The exponential has value $e^0=1$. Its local rule multiplies the input tangent by that value, so the tangent of $f$ is $1\cdot 1=1$. The closed form checks the result: $\partial f/\partial {\theta }_1={\theta }_2\exp ({\theta }_1{\theta }_2)=1\cdot 1=1$. The exponential is one of the primitives whose backward rule reads its own output, which is why section 18.4 says the trace must store more than the intermediate inputs. This forward pass never formed the derivative expression ${\theta }_2\exp ({\theta }_1{\theta }_2)$; it applied one local rule at each operation.

The output has value $f=3^2\cdot 2=18$ and adjoint 1. The product $f=s{\theta }_2$ sends its adjoint to each parent multiplied by the other factor: $s$ receives $1\cdot 2=2$, and ${\theta }_2$ receives $1\cdot 9=9$. The square $s={\theta }_1^2$ sends its adjoint to ${\theta }_1$ multiplied by the local slope $2{\theta }_1=6$, so ${\theta }_1$ receives $2\cdot 6=12$. The two slopes are 12 and 9, and one backward pass produced both. The formula agrees: $\partial f/\partial {\theta }_1=2{\theta }_1{\theta }_2=12$ and $\partial f/\partial {\theta }_2={\theta }_1^2=9$. Forward mode needed two passes for the same two numbers, one per input. Reverse mode needed one pass because the function has one output, and the count follows the output side.

The forward values are $a=3\cdot 3=9$, $t=9+9=18$ and $L=18\cdot 18=324$. The closed form is $L=(2x^2{)}^2=4x^4$, so $dL/dx=16x^3=16\cdot 27=432$. The backward sweep agrees. Seed $L$ with 1. The square $L=t\cdot t$ has local slope 18 on each of its two edges, so $t$ receives $1\cdot 18+1\cdot 18=36$. The sum $t=a+a$ has slope 1 on each edge, so $a$ receives $36+36=72$. The square $a=x\cdot x$ has local slope $x=3$ on each edge, so $x$ receives $72\cdot 3+72\cdot 3=432$. The graph has six edges because $x$, $a$ and $t$ are each read twice, and the sweep performs exactly one local multiplication per edge. Six edges, six multiplications, one answer.

For $L=16$ the logarithm is ${\log }_{2}16=4$. Recursive halving makes $(16/2)\cdot 4=8\cdot 4=32$ calls and holds 4 slots. Storing everything makes $16-1=15$ calls and holds 16 slots. The work ratio is $32/15=2.1333$, which rounds to 2.13. The memory ratio is $16/4=4$. Recursive halving therefore does about twice the block evaluations of storing everything and holds a quarter of the values. The gradient is the same in both schedules, provided a recomputed block repeats its first run. The trade is the one section 18.5 prices: memory falls from linear in the depth to logarithmic, and computation rises by a factor that grows with the logarithm of the depth.

Symbolic differentiation fails because the program has no closed formula: the loop trip count and the branch depend on the input, so there is no fixed expression to differentiate. Numerical differentiation can run the program and return an estimate, but its accuracy is limited by subtracting two close values, and the exercise asks for exactness. Automatic differentiation reads the operations that ran, applies one local rule per operation, and returns exact values for that run. A branch costs what the branch did, and a loop costs what the loop ran, so the price follows the program rather than the length of a formula. Option (c) is correct. The word "exactly" belongs to the code, not to the mathematical function: section 18.1's limitation says a branch or a corner can make the code's derivative differ from the derivative of the function the programmer meant.