The derivative as a best linear map

Here is a curve with no obvious slope: $y=x^2$ at the point where x $=3$. Pick a second point a little to the right and measure the straight line through the two. With the second point 0.1 away, the line has slope 6.1; at 0.01 away the slope is 6.01, and at 0.001 away it is 6.001. The numbers are converging on 6.

That convergence answers the question this section asks: what is the slope of a curved thing? The slope of a curved thing at a point is the number that the slopes of these straight lines approach as the second point slides toward the first. Each straight line is a secant, and its slope is the rise divided by the run between the two points. The run shrinks toward zero while the ratio settles on a value, and that value is the local rate of change. !Four chords through the point $(3,9)$. At a gap of 0.5 the slope is 6.5, at 0.1 it is 6.1, at 0.01 it is 6.01, and the tangent is 6. The lower panel plots the quotient against the gap: the leftover gap is the whole distance between chord and tangent, so the numbers close on 6 without reaching it at any nonzero gap. Status: re-simulated from $f(x)=x^2$.

Written in symbols, the secant slope is the difference quotient. The derivative is the limit of that quotient: !The absolute value is continuous at zero and has no slope there. Approach from the left and every secant has slope $-1$; approach from the right and every secant has slope $+1$. A derivative exists only when the two approaches agree, and here they never do. Status: schematic.

That line defines the derivative as the limit of the ratio: the prime marks the derivative, and ${\lim }_{h\to 0}$ says the ratio is followed as the gap $h$ shrinks toward zero. The ratio itself is called a difference quotient [S:22:81].

For the square the quotient simplifies. Expanding $(3+h{)}^2$ gives $9+6h+h^2$, so the ratio is $6+h$ and the limit is 6. In general $(x+h{)}^2=x^2+2xh+h^2$, the ratio is $2x+h$, and the derivative is $2x$. The single leftover $h$ is the whole difference between the secant and the tangent.

The same limit has a physical reading. If $s(t)$ is the position of a particle at time $t$, the difference quotient is the average velocity over the interval, and the derivative is the instantaneous velocity at the point [S:121:118]. The average is easy to measure, and the instantaneous value is what a speedometer displays.

The title of this section names the second reading, which matters more for machine learning. Among all straight lines through the point, the tangent line is the one that touches the curve there. The gap between the line and the curve shrinks faster than the distance from the point, so no other line stays closer. That line exists exactly when the derivative does, and its slope is that derivative. The equation of the line is the linear approximation [S:22:84]:

A derivative is therefore not only a number but a small linear map: the rule that multiplies a small change in the input by the slope to give the change in the output. It is also the best straight-line stand-in for the curve near that point.

Several notations name the same limit. A prime writes $f^{\prime }(x)$, a ratio of small differences writes $dy/dx$ for the same object, and plain prose calls it the rate of change. This book uses the prime in formulas and the word rate in sentences, and either one names the same object.

Differentiating a second time gives the second derivative, which records how the slope itself changes [S:22:81]. A positive second derivative means the slope is increasing and the curve bends upward; a negative one means the opposite. That quantity is what the next section adds to the straight line, and it is the reason a line runs out of accuracy.

Worked example. Use the tangent at $x=3$ to estimate $3.1^2$. The line is $T(x)=9+6(x-3)$, so $T(3.1)=9.6$ while the true value is $9.61$: the miss is $0.01$, the square of the gap. At a gap of $0.01$ the miss is $0.0001$, and section 10.3 supplies the general size.

Two warnings follow from the definition. A derivative needs the same limit from both sides, so a corner has none: the absolute value has slope $-1$ on the left of zero and $+1$ on the right [S:22:87].

Differentiability is also stronger than continuity. A function is differentiable at a point when the derivative exists there, and every such function is continuous at that point. The converse fails, and the absolute value is again the example [S:22:84-85].

The mean value theorem is the exact version of the secant picture: on the run from $3$ to $3.1$ some interior point has slope exactly 6.1, the secant slope the section opened with [S:22:91]. That is why a bound on the derivative controls the change across the whole interval.

Limitation. A derivative describes one point and its immediate neighbourhood, not the curve as a whole. Two functions can share a value and a slope at a point and diverge quickly afterwards, so a linear approximation is trustworthy only over a range you have bounded. The next section supplies the bound, by adding curvature to the straight line.

The formal statement. A function $f$ is differentiable at an interior point $a$ when the difference quotient $(f(a+h)-f(a))/h$ has a limit as $h\to 0$ [S:22:81]. The limit is $f^{\prime }(a)$. The same condition has a second form: there is a number $m$ and a function $E$, defined near zero, with

$$f(a+h)=f(a)+mh+E(h)h,\underset{h\to 0}{\lim }E(h)=0.$$

That line says the curve equals its value plus a straight-line term plus an error smaller than $h$. The two forms are equivalent, and $m=f^{\prime }(a)$ [S:22:84].

The derivation. Suppose the quotient tends to $m$. Set $E(h)$ equal to the quotient minus $m$ for $h\ne 0$, and set $E(0)=0$ [S:22:84]. Then the identity holds and $E$ tends to zero. In the other direction, subtract $f(a)$, divide by $h$, and the leftover term is $E(h)$, which vanishes in the limit.

Why the linear-map reading matters. In several variables the same clause reads $f(w+v)=f(w)+\partial f(w)[v]+o(\Vert v\Vert )$, where $\partial f(w)$ is a linear map [S:72:36]. The number $m$ becomes a matrix, the Jacobian. Calling the derivative the best linear map is therefore the version that survives a change of dimension; the quotient is the one-dimensional special case.

The edge case. Continuity does not supply a derivative. The absolute value is continuous at zero and has no derivative there, because the two one-sided quotients disagree [S:22:84-85]. Worse, almost every Brownian path is continuous everywhere and differentiable nowhere; the exceptional set has probability zero [S:87:416]. L1 opens with a curve of no obvious slope and then finds one, which suggests that a slope always exists once you look. It does not.

What L1 hid. L1 moved between "a number" and "a small linear map" without saying when the two coincide. They coincide for one input and one output. For many inputs the derivative is a list or a matrix, and the slope depends on the direction of the step. L1's picture was safe because every example had one input.

Where the analogy breaks. "Best straight-line stand-in" means the error is $o(h)$ as $h\to 0$. It does not mean the line has the smallest error at a fixed gap. At a fixed gap another line can fit better, and the derivative says nothing about that line. The word "best" is a statement about a limit, not about a ruler.

Original sources. The derivative as the limit of the difference quotient is Definition 2.3.1 in Trench [S:22:81]. The same book gives the equivalent error form, with a function that vanishes at the point [S:22:84]. The several-variable version, with differentiability defined by a linear directional derivative and a vanishing normed error, is Definition 2.6 in Blondel and Roulet [S:72:36].

An open question. A trained network is differentiable almost everywhere, because it is locally Lipschitz and Rademacher's theorem applies [S:72:70]. At the exceptional points no derivative exists, and the Clarke subdifferential supplies a set of generalized gradients instead [S:72:71]. Software returns one number there. The field has not settled whether that convention changes what training finds. A convention that ignores the kink is cheap, and a convention that averages the one-sided slopes is also cheap, yet the two take different steps from the same point. Evidence that would settle it: the rate at which iterates land on the exceptional set, measured during training, compared across conventions with the same seeds and the same data order.

A frontier result. Reverse-mode automatic differentiation computes the derivative of a program by composing the Jacobians of its steps [S:72:73]. It is the same product of local rates that you met in one variable, applied to a graph rather than a line. For nonsmooth programs, conservative gradients extend the chain rule to a class of functions where the classical derivative fails [S:72:72]. That is the current edge of the subject.

What to read next. Read Trench §2.3 for the definition, the tangent-line lemma and the proof that differentiability implies continuity [S:22:81]. Read Blondel and Roulet Chapter 2 for the Fréchet form and §2.7 for Rademacher and Clarke [S:72:70]. Read the appendix of the diffusion book for continuous nowhere-differentiable paths as a concrete warning [S:87:416]. The three sources use different notations for the same limit, so read the definitions side by side.