Checking a derivative

You have written down that the derivative of $x^3$ at x $=2$ is 12. A formula on paper is a claim, and claims about derivatives can be checked with arithmetic. Step 0.01 to the right and compute the rise over the run: the cube of 2.01 is 8.120601, so the quotient is 12.0601. Step 0.01 to each side instead and divide by the whole width: the quotient is 12.0001. Both estimates sit near 12, and the second is far closer.

This is the answer to the question this section asks — how do you know the formula is right — and it is a habit rather than a theorem. A finite-difference check replaces the limit with a small but nonzero step, computes the resulting quotient, and compares that number with the formula. The gap between the two is the truncation error, and its size tells you whether the formula is right and how much the check can see. !The measured error of the two checks on x cubed at 2, against the step. The forward difference falls along a line of slope one and the central difference along slope two, so halving the step divides their errors by two and by four. The central curve reaches lower and turns upward sooner, near two times ten to the minus six, where subtracting two nearly equal numbers begins to cost more than the smaller step gains; the forward curve turns near two times ten to the minus eight. Status: re-simulated.

The simplest version, a forward difference, steps only forward:

That line says the derivative is roughly the rise over a short run to the right, and the approximation improves as the step shrinks [S:22:113]. Taylor's theorem supplies the price: expanding once more gives $f(x+h)=f(x)+h{f}^{\prime }(x)+h^2{f}^{\prime \prime }(\xi )/2$ for some point $\xi$ between $x$ and $x+h$, so after dividing by $h$ the error is $h{f}^{\prime \prime }(\xi )/2$. Its size is at most half the step times the largest absolute value of the second derivative on the interval, and the one half is the $1/2$ of the quadratic term. Halve the step and the bound halves with it, which is slow but predictable.

A central difference steps both ways and divides by the full width:

That line says the derivative is roughly the rise across a short symmetric interval divided by its width. Its error is proportional to the square of the step rather than the step itself, so halving the step divides the error by four [S:72:192]. That is why the estimate 12.0001 beats 12.0601 at the same step size, and why central differences are the default for checking work by hand.

Worked example. Check the claim for $x^3$ at $x=2$ without a calculator beyond arithmetic. The cube of 2.01 is 8.120601 and the cube of 1.99 is 7.880599. The forward quotient is $(8.120601-8)/0.01$, which is 12.0601.

The central quotient is $(8.120601-7.880599)/0.02$, which is 12.0001. The exact derivative is $3x^2$, which is 12 at $x=2$, and the central estimate is wrong by about one part in a hundred thousand.

In a program the same comparison runs at scale. A gradient implementation returns a list of numbers, and the check computes a finite-difference estimate for each entry and compares the two lists [S:27:155].

Software usually reports a relative error, the absolute difference divided by the size of the value being checked, so that a large gradient and a small one are judged on the same scale. A tolerance of a millionth is a common bar for a correct implementation. The cost is one extra evaluation of the function per entry, which is why a check runs on a few points rather than on the whole dataset.

The comparison is evidence, not proof. A check tests the points you feed it, and a formula can be right at those points and wrong elsewhere.

It is also sensitive to the step: too large and the truncation error dominates, too small and the subtraction of two nearly equal numbers destroys the digits, a subject the chapter on floating point takes up [S:72:191]. The useful range of steps is a compromise between those two failures.

Limitation. A finite-difference check compares numbers, not meanings. It catches a wrong sign, a missing factor and a coding slip, and it passes happily when the formula is right but the assumption behind it is wrong. The source is honest about the strength of the conclusion: if the error is small, the implementation is probably correct [S:27:155]. Probably is the right word, and a second test at a different point costs almost nothing.

The formal statement. Let $f$ have a second derivative on an interval containing $x$ and $x+h$. The forward difference obeys

$$\left|\frac{f(x+h)-f(x)}{h}-f^{\prime }(x)\right|\le \frac{M_{2}h}{2},$$

where $M_2$ bounds $|f^{\prime \prime }|$ on that interval [S:22:113]. That line says the error of the forward difference is at most half the step times the largest curvature. The central difference obeys

$$\frac{f(x+h)-f(x-h)}{2h}=f^{\prime }(x)+O(h^2),$$

because the even-order terms cancel [S:72:192]. That line says the symmetric quotient has an error of order the step squared.

The derivation. Expand $f(x+h)$ and $f(x-h)$ by Taylor's theorem to the third order, then subtract. The terms of order zero and two cancel, the terms of order one add, and a term of order $h^3$ remains [S:72:192]. The difference is $2h{f}^{\prime }(x)$ plus that remainder. Divide by $2h$. The constant in the resulting bound involves the third derivative, which the check does not know. The omitted step is the bound on the round-off term, which depends on the machine and on the implementation.

The edge case. Take $f(x)=|x|$ at zero. The central quotient is $(|h|-|-h|)/(2h)=0$ for every step, so the check returns zero and would confirm the claim $f^{\prime }(0)=0$, although no derivative exists there. A check confirms a number at a point. It does not confirm differentiability at that point.

The round-off side. The computed function is rounded to machine precision, so the numerator carries an error of order $ϵ$ and the quotient carries a term of order $ϵ/h$ [S:72:191]. Truncation falls as $h$ shrinks and round-off rises, so a best step exists between the two regimes [S:72:192]. The two terms pull in opposite directions, so the total error has a minimum rather than a limit of zero. L1 said that halving the step halves the error, which is the truncation term alone, valid while that term dominates.

What L1 hid. L1 treated the step as a free parameter chosen small. It is not free: below a threshold, subtracting two close numbers destroys the digits. The formal statement replaces "small" with two named terms and a range where each dominates.

Where the analogy breaks. L1 said the check is evidence, not proof. The formal version says more: the check tests one point and one implementation. A relative error below a threshold is a convention, and the source says only that the implementation is probably correct [S:27:155]. The word "probably" carries no number.

Original sources. The forward-difference error bound, with the second-derivative constant, is derived in Trench §2.5 [S:22:113]. The systematic treatment of forward, backward and central differences, with their truncation orders, is Chapter 7 of Blondel and Roulet [S:72:192]. The gradient-check procedure, with a relative-error threshold, is the remark in MML §5.2 [S:27:155].

An open question. A finite-difference check tests the points you give it. Certifying that an implementation is correct on every input is a different problem, and no method scales to a network with millions of parameters. A single check at a random point is cheap; a check that covers a region needs a bound on how fast the derivative varies, which is the same object the difference is trying to estimate. What is unsettled is how far a small number of checks can be trusted. Evidence that would settle a case: a machine-checked proof of the derivative of the program, or a symbolic identity check over the whole input domain, together with a test that the checker itself is sound.

A frontier result. Where every operation in the program is defined on the complex numbers, the complex-step derivative evaluates $f$ at a complex step and reads the derivative from the imaginary part. It needs a single complex function call, roughly twice the cost of a real one, and avoids the subtraction of two close numbers, so its round-off error is smaller than that of a central difference [S:72:195]. It does not apply to programs containing comparisons or other operations that are not defined on the complex numbers. Finite differences cost a number of calls proportional to the dimension, which is why they serve as tests rather than as the training mechanism [S:72:197].

What to read next. Read Trench §2.5 for the error bound and the forward-difference derivation [S:22:113]. Read Blondel and Roulet Chapter 7 for the three difference schemes and their complexity [S:72:191]. Read the remark in MML §5.2 for the form of a practical check and for its wording about probability [S:27:155]. The three schemes share the same step-size trade-off, and only their truncation orders differ.