Where autodiff lies

At x = 0 the function max(0, x) has no derivative. From the left its slope is 0, from the right it is 1, a library returns 0, and a centred difference returns 0.5. None of those four numbers is a bug. Each answers a different question about the same corner.

The corner is a kink: a point where the two one-sided slopes disagree. A function whose graph is straight between its corners is piecewise linear, and its derivative exists everywhere except at those corners. A corner has no width, so a random input almost never lands on one. When training does land on one, the number it receives is a choice rather than a fact. !At the corner the left slope is 0 and the right slope is 1. A library returns 0, the left-hand value by convention; a centred difference with a step of 0.1 draws the chord from $(-0.1,0)$ to $(0.1,0.1)$ and returns 0.5, the average of the two one-sided slopes. The symmetric derivative is 0.5, and the derivative itself does not exist. Status: schematic.

Worked example. Centre the quotient on the kink with step 0.1. The value at 0.1 is 0.1, the value at −0.1 is 0, and the quotient is (0.1 − 0)/0.2 = 0.5. The two one-sided slopes are 1 and 0, and 0.5 is their mean. That mean is the symmetric derivative, not the derivative. !The rounded value is piecewise constant, so its slope is zero wherever it is defined. Every parameter whose only path to the loss crosses that zero receives no update, and the ghosted boxes keep their initial values. The run continues and the loss can still fall along other paths, which is why the missing capacity is invisible. Status: schematic.

The formula demonstrates why the answer cannot be anything else. With a step written Δ,

That line says a centred difference at a corner returns the average of the two one-sided slopes, whatever the size of the step.

A library must choose a number anyway. Dive into Deep Learning documents the left-hand slope, 0, for its rectified linear unit, the primitive max(x, 0) [S:69:212]. A library that implements the identical function as a maximum of two arguments breaks the tie by its own rule. Blondel and Roulet write that "in case of ties, we break them arbitrarily" [S:72:159], so a rule that favours the argument with the unit slope returns 1, while an even split returns 0.5.

The convention is a convention: a rule adopted so the code returns something, not a value derived from the function. A one-sided derivative would be defensible too, and so would the average. The point is to know which one you are getting.

The same quiet failure appears in functions that only compare. A step function, an argmax or a rounding operation is piecewise constant, so its slope is zero almost everywhere and undefined at the jumps [S:72:162]. A zero gradient is worse than an error. An error stops the run, while a zero enters the update and is discovered a week later.

Kinks are common in practice. A rectified linear unit is max(0, x) applied to every coordinate, a pooling layer takes a maximum over a window, and a clipped loss stops changing outside a range. Each introduces a corner, and a large network can contain millions of them. The corners are rare as inputs rather than rare in the code, because a random value almost never lands exactly on one [S:69:212].

The damage increases with the scale of the model. Every parameter whose only path to the loss crosses a zero derivative receives no update at all. A piecewise-constant feature, such as a bucket index or a comparison flag, therefore removes every parameter behind it from the training step. The run continues, the loss falls for a while, and the missing capacity is invisible.

A tie makes the choice visible. When two arguments of a maximum are equal, a rule must decide whether the derivative goes to the first, the second, or half to each. The value of the function is the same under all three rules, so no test on values can tell them apart. Only a test on the gradient can.

A kink also breaks the second derivative. The slope of max(0, x) jumps from 0 to 1 at the origin, so the derivative is itself a step [S:74:118]. Differentiate that step and the result is zero away from the corner and undefined at it, which is not a curvature any second-order method can use. The failure is inherited from the first derivative, not introduced by the second.

The cost is paid in modelling rather than in arithmetic. A model built from kinks fits a piecewise-linear surface, and at a corner it receives one of several defensible gradients. For a convex function, a subgradient is the slope of a line that touches the graph and stays below it, and the library reports one member of that set. The honest response is to learn which member, and to keep the input away from the corner when the answer matters.

The practical diagnostic is to move the input away from the corner and compare the automatic gradient with a centred difference. Away from the kink the two agree to several digits, and at the kink they disagree by design. A disagreement away from every kink is a bug in the derivative code. A disagreement at a kink is the convention doing its job.

This is the chapter's theme in its sharpest form. The computer differentiates the operations the program ran, and a program built from max, round and compare has a derivative built from steps and zeros [S:69:212]. The mathematical function the author had in mind may be perfectly smooth, and the program is still not.

Limitation: Automatic differentiation cannot detect that the program it differentiated is not the function you meant. A comparison, a rounding or a max with a tie is still an operation, and it still returns a number. A zero gradient is a valid answer to a question you did not intend to ask.

The formal statement. Let a program denote a function $f$, and let $J(x)$ be the derivative the engine returns for the trace at $x$. Two claims follow.

1. Where $f$ is differentiable at $x$ and the trace executed at $x$ is the one the program executes on a whole neighbourhood of $x$ — no predicate changes value at $x$ and no two branches meet there — $J(x)=f^{\prime }(x)$.
2. Where $f$ is not differentiable at $x$, $J(x)$ is a convention that depends on the program, not on the function. Precisely: two programs exist with the same $f$ everywhere and different $J$ at some point.

Claim 1 holds when every primitive on the executed path is differentiable at the value it received and the branch taken at $x$ stays the branch on a neighbourhood of $x$. Claim 2 needs no hypothesis; it is a statement about two programs.

The derivation of claim 1. The trace is a chain of primitives, and 18.2 and 18.3 show that the two sweeps compute the derivative of that chain. If the composite is $f$ and the composite is differentiable at $x$, the chain rule gives $f^{\prime }(x)$ — provided the trace at $x$ is also the trace on a neighbourhood. That proviso is not implied by differentiability of $f$. With the left-hand slope at zero [S:69:212], the program $x\mapsto \mathrm{relu}(x)-\mathrm{relu}(-x)$ computes the identity, whose derivative at 0 is 1, and its trace returns $0-0=0$. The engine returns the derivative of the branch the run took and does not propagate the predicate's own derivative [S:72:154].

The derivation of claim 2. Take two programs that compute the same function. The first computes $\max (x,0)$ and breaks the tie in favour of the first argument, which is the rule that returns the gradient of a chosen argmax branch [S:79:189]; at zero it returns 1. The second computes $\max (0,x)$, the same function with the arguments in the other order, and takes the left-hand slope at the tie, which is 0 [S:69:212]. The two programs return different numbers at the only point where it matters, so the returned value belongs to the primitive and its tie rule, not to the function. The same point appears with $\max (x,-x)$ against $|x|$: an even split at the tie gives $0.5\cdot 1+0.5\cdot (-1)=0$, while the left-hand slope of $|x|$ is $-1$, so those rules also disagree. Neither number is the derivative, because no derivative exists.

The edge case. A selection. A program that returns the larger of two computed values has a subdifferential at a tie that contains both one-sided slopes, so some value is available. A program built on argmax is worse: the map is piecewise constant, so its Jacobian is zero almost everywhere and undefined at a tie [S:72:162]. A parameter that only changes which branch is selected therefore receives a zero gradient, and training stalls with no error. The zero is a valid subgradient of a function that genuinely does not depend on the parameter away from the tie.

A second edge case is the removed edge. Setting $u$ equal to a computed $y$ but detached, then forming $z=ux$, returns $u$ as the derivative in $x$, because no edge links $u$ to whatever produced it [S:69:102].

What the one-paragraph version hides. "Where autodiff lies" is personification, and the precise statement is duller and more useful: the engine answers a well-posed question exactly, and the question is about the trace. The gap between the trace and the function you meant has three sources: a kink, a removed edge, and a branch that the particular run took.

Where the analogy breaks. A lie needs a speaker who knows the truth. Nothing in the engine compares the trace with your intention, so it cannot lie. It can only answer the trace's question, and the cost of asking the wrong question is silence.

Original sources. The control-flow chapter of Blondel and Roulet treats if-else, conditionals, argmax and loops one at a time, and states for each what the Jacobian is and where it stops being informative [S:72:160]. Shalev-Shwartz and Ben-David define the subgradient and compute the subdifferential of the absolute value, which is the standard repair at a kink [S:79:189]. Dive into Deep Learning shows detach as a program that removes an edge from the recorded graph [S:69:102].

An open question. What a library should return at a point where the derivative does not exist, so that training stays consistent with the objective the user meant. The available answers disagree: a minimum-norm subgradient, a one-sided slope, a continuous relaxation, or a surrogate chosen by hand. The choice is made per primitive by the library, not per problem by the user, and no test inside the program can tell whether it was the right one.

What would settle it: a benchmark in which the optimum lies on a kink, trained under each convention at matched compute, with the returned value checked against the subdifferential set at every step. The evidence must include a case where the conventions give different trained parameters, otherwise the question was never exercised.

A frontier result. Continuous relaxation. Replacing a hard comparison with a sigmoid makes the predicate continuous, so the Jacobian is non-null everywhere and gradients cross the branch [S:72:155]. Replacing argmax with softargmax does the same for a many-way selection [S:72:162]. The price is stated with the result: the expectation over branches requires evaluating every branch, not the one the run would have taken [S:72:162]. That is the trade the whole chapter is about, moved from memory to computation.

What to read next. Blondel and Roulet chapter 5 for the taxonomy of control flow and the relaxation of each operator [S:72:160]. §14.2 of Shalev-Shwartz and Ben-David for subgradients, the differential set and subgradient descent [S:79:190]. §2.5.3 of Dive into Deep Learning for detach, and §2.5.4 for the control-flow case where the graph is only known after the run [S:69:103].