Taylor approximation and curvature

Take $e^{0.1}$, where $e$ is the fixed number near 2.718 that describes growth at a constant relative rate; no school formula gives $e^{0.1}$ directly. A calculator returns 1.1052. You can reach 1.105 with two multiplications and two additions: square 0.1, halve it, add 1, and add 0.1. The short formula is wrong by less than 0.0002, and it needs no exponential at all.

That formula is the answer to the question this section asks: how do you replace a curve with a polynomial? A Taylor polynomial replaces a curve with a polynomial that matches the function's value, slope and higher derivatives at one chosen point. Each extra term matches one more derivative at that point, which widens the stretch of curve the polynomial tracks before it peels away; over a wide interval a higher degree can still do worse, as the limitation below shows. Polynomials are cheap to evaluate, which is why considerable effort goes into using them to stand in for more complicated functions [S:22:106]. !The constant 1, the line $1+x$, the quadratic $1+x+x^2/2$ and the cubic, against the exponential. At 0.1 the quadratic is wrong by 0.00017; at 1 it is wrong by 0.22. Each extra term matches one more derivative at zero, which widens the stretch of curve the polynomial tracks before it peels away. Status: re-simulated from the exponential and its Taylor polynomials.

The construction is mechanical once you fix the point $a$: !Partial sums of $1+x+x^2$ and so on, against the number of terms. At $x=0.5$ the error halves with each term and the sums close on 2. At $x=2$ the error doubles with each term while the true value is $-1$, so every added term moves the sum further from the answer. The radius of convergence is the boundary between those two behaviours. Status: re-simulated from the geometric series.

That line is a sum, written with the Greek sigma: $k$ runs from 0 to $n$. Each term is the $k$-th derivative of the function at the point, written $f^{(k)}(a)$, divided by $k!$ — the product of the whole numbers from 1 to $k$ — and multiplied by $(x-a{)}^k$. The coefficient of each power is that quotient, and it is exactly what makes the polynomial's derivatives agree with the function's [S:27:148]. The zeroth term is the value, the first term adds the slope, and the second term adds curvature, which is the rate at which the slope itself changes.

Write the polynomial as T_n, with the subscript giving its degree, the highest power it contains. The coefficients are forced, not chosen. A degree-n polynomial has n+1 coefficients; asking its value and its first n derivatives at the point to equal the function's gives n+1 linear conditions, and they have exactly one solution [S:22:107]. That solution is unique: no other polynomial of degree at most n approximates the function that closely near the point [S:22:108].

The factorial in the denominator is not decoration. Differentiating $x^k$ exactly k times leaves k factorial, and every later derivative is zero, so dividing by k factorial is what makes the k-th derivative of the polynomial equal the k-th derivative of the function [S:22:107]. Without that divisor, each term would contribute several copies of the same derivative and every coefficient after the first would come out wrong.

Worked example. Expand $e^x$ at $a=0$, where every derivative equals 1. The degree-two polynomial is $1+x+x^2/2$. At $x=0.1$ it gives $1+0.1+0.005$, which is 1.105.

Taylor's theorem bounds the error by the next term, evaluated at some unknown point between 0 and 0.1. That bound is smaller than 0.0002, so the three-term answer is correct to three decimal places [S:22:112].

The remainder is the gap between the polynomial and the function, and the theorem states its size rather than hiding it. For a smooth function the remainder after n terms looks like the next term of the expansion, with the derivative evaluated somewhere in the interval. That is why the error shrinks so fast near the point: the powers of the distance do the work.

Curvature is the new ingredient, and it explains why a straight line is not enough. The tangent line matches the value and the slope, so its error is proportional to the squared distance. The quadratic term matches the second derivative too, so its error is proportional to the cubed distance. Each step down this ladder multiplies the accuracy near the point, and the price is one more derivative to compute.

The expansion can also be exact. For a polynomial of degree four, the Taylor polynomial of degree four about any point reproduces it with no error, because every derivative above the fourth vanishes [S:27:149]. Approximation is needed only when the function is not already a polynomial of low degree.

Choosing how many terms to keep is a trade between accuracy and work. Each extra term needs one more derivative at the point and one more multiplication at each evaluation. The remainder bound is the stopping rule: keep terms until the bound falls below the tolerance you require, and stop there. In the exponential example, three terms already meet a tolerance of 0.0002.

Limitation. A Taylor polynomial is a local object, and its accuracy is a claim about an interval, not about the whole line. The geometric series $1+x+x^2+\cdots$ converges to $1/(1-x)$ when the absolute value of $x$ is below 1, and diverges at $x=2$ — its partial sums grow without a limit — while the function $1/(1-x)$ equals $-1$ there [S:5:636][S:122:479]. Away from the expansion point the error grows, and a polynomial of high degree can be worse than a low one.

The formal statement. Let $f$ have $n+1$ derivatives on an open interval $I$ containing $a$. For each $x$ in $I$ there is a number $c$ between $a$ and $x$ with

$$f(x)=T_n(x)+\frac{f^{(n+1)}(c)}{(n+1)!}(x-a{)}^{n+1},$$

where $T_n$ is the Taylor polynomial of degree $n$ [S:22:112][S:122:511]. That line says the error is the next term of the expansion, with its derivative evaluated at an unknown interior point. The coefficient of the term of degree $k$ is $f^{(k)}(a)/k!$ [S:22:107]. Those coefficients are forced: $n+1$ derivative conditions fix $n+1$ coefficients, and the polynomial is unique [S:22:108].

The derivation. Differentiate $T_n$ exactly $r$ times at $a$. Every term of degree below $r$ dies, every term above $r$ still carries a power of $(x-a)$, and the term of degree $r$ contributes $r!$ times its coefficient. Setting the result equal to $f^{(r)}(a)$ gives the coefficient. The remainder comes from applying Rolle's theorem to an auxiliary function built from $f$ and $T_n$ [S:122:511]. I omit that construction; it is a page of algebra, and the statement is what later sections use.

The edge case. The theorem needs $f^{(n+1)}$ on the interval. With only $n$ derivatives at the point you get the weaker limit $f(x)-T_n(x)=o((x-a{)}^n)$ and no bound at a fixed $x$ [S:22:107]. Convergence of the Taylor series is a separate question: the series equals $f$ exactly when the remainder tends to zero [S:122:510]. A series can converge on an interval and still fail that test, so convergence of the series is not the same claim as equality with the function.

What L1 hid. L1 said that calculators and networks evaluate functions this way. The safe part is the principle: a smooth function is locally a polynomial plus a small error. Real software uses a fixed degree and a reduced range, because a Taylor polynomial is accurate near $a$ and poor far from it. L1's sentence was an illustration of the principle, not a claim about an implementation.

Where the analogy breaks. "Each extra term buys one more derivative of agreement" is exact at $a$ and only there. Away from $a$ the gain depends on the interval and on the size of the next derivative. A high-degree polynomial can be worse than a low-degree one on a wide interval, which is why the remainder bound, not the degree, is the stopping rule.

Original sources. Taylor's theorem with the Lagrange remainder is Theorem 2.5.4 in Trench [S:22:112] and Theorem 6.7 in the OpenStax calculus volume [S:122:511]. The uniqueness of the Taylor polynomial is stated in Trench and left as an exercise [S:22:108].

An open question. The remainder bound needs a bound on $f^{(n+1)}$ over the interval, and for a trained network that derivative is not available in closed form. How many terms are needed for a stated accuracy away from the expansion point therefore has no practical general answer. The bound is also pessimistic, because it uses the worst point of the interval rather than the point where the polynomial is evaluated. A bound that is loose by a constant still answers the question the section asks; a bound that is loose by a factor growing with the interval does not. Evidence that would settle a given case: a bound computed from the model's higher derivatives, checked against the measured error on inputs the bound was not fitted to.

A frontier result. Newton's method replaces the linear model by the quadratic Taylor model and steps to its minimum. Its convergence is quadratic near a solution, which is the payoff of keeping the second-order term [S:44:510]. The price is the Hessian: forming and storing it, and solving a linear system for each step, which is cubic in the number of parameters unless structure is exploited [S:44:510].

What to read next. Read Trench §2.5 for the theorem and its use on forward differences [S:22:112]. Read OpenStax §6.3 for the remainder and the convergence criterion [S:122:510]. Read the unconstrained-minimization chapter of Boyd and Vandenberghe for what the second-order term buys and what it costs [S:44:509]. Then compare the two error bounds on one example and note where each is loose.