Limits and continuity

Take a number, halve it, add one. Start at 1 and the rule gives 1.5, then 1.75, then 1.875, then 1.9375. Each successive answer sits closer to 2 than the previous one, and the distance to 2 halves at every step: 1, then 0.5, then 0.25, then 0.125.

That shrinking distance is what mathematicians mean by arbitrarily close: choose any tolerance, however tight, and after enough steps the distance stays below it. The value 2 is the limit of the run, and the whole chapter rests on this single notion. Every slope and gradient that follows is a limit of the same kind, computed for a different mathematical object, so the measurement of change begins here. !Each term of $x_{k+1}=x_k/2+1$ sits closer to 2 than the one before it. The number line breaks after the early terms so that both tolerance bands appear at scale; a band of half-width 0.01 is entered at step seven and never left, and a band of 0.001 needs ten steps. The lower panel plots the distance on a log scale, where every step halves it and each tolerance is a flat line the run crosses once. Status: re-simulated from the iteration rule.

The run is a sequence: one value for each whole-number step. Its rule fits on one line. The letter x stands for the current value, and the subscript counts the step:

That line says each new value is half the previous value plus one, and the values creep up on 2 from below. What makes the run predictable is the factor $1/2$: each step multiplies the distance to 2 by that factor, so the distance after $k$ steps is known before the run starts. A rule with that property is a contraction, and a contraction is what turns a tolerance into a step count.

The general statement, with its closed interval and its one uniform factor, is in the Deepen below, together with the two caveats: a factor below one taken pair by pair is not enough, and neither is an open interval. For this run the factor is $1/2$, and the worked example shows what that buys. The bracketed tag names the source and the page it supports.

Worked example. Specify a tolerance of 0.01, which fixes the precision you demand. Each step halves the distance, which starts at 1, so after k steps it is 1 divided by 2 to the power k. At six steps it is 0.0156, too wide, and at seven steps it is 0.0078, which passes. Ten iterations give 0.00098, so the same rule meets a tolerance ten times tighter after three more steps.

The computation requires a single division per iteration, and the guarantee is a bound, a statement that the distance stays below a named number from some step onward. That shape, a tolerance followed by a step count, is the entire content of the definition. It is also the reason a limit is a statement about a tail: the initial values can be as erratic as they please, provided the run eventually stays inside every band you draw around the limit.

Picture the tolerance as a horizontal band around 2, with edges at 2 minus the tolerance and 2 plus the tolerance. The definition says the run enters that band and never leaves it, and the picture transfers to a function, a rule that assigns an output to each input, without much change. There the band runs across the output axis and the interval runs along the input axis, and the definition asks for an input interval narrow enough that the graph above it stays inside the band.

The definition of a limit at a point has the same shape, with one detail that surprises people. The limit does not involve the value of the function at that point, and the function need not be defined there at all [S:22:42]. What matters is the approach: an algebraic expression can be undefined at a point and still possess a limit there, because the limit records where the values are heading. The plain reading is old: the output can be made as close to the limit as you wish by taking the input close enough [S:22:41].

A concrete case shows the point. The expression $(x^2-9)/(x-3)$ is undefined at $x=3$, because the bottom is zero. At 2.9 it equals 5.9, at 2.99 it equals 5.99, and at 2.999 it equals 5.999. The limit is 6, even though the expression has no value at 3, and the cancellation that produces 6 becomes apparent once you factorise the numerator.

The same test applies to a function rather than a sequence: a function is continuous at a point when its limit there equals its value there [S:22:62]. Inputs that approach the point give outputs that approach the value with no jump at the end, so continuity closes the gap between where the function is heading and where it sits. !Continuity reads the same definition sideways. Pick a band of half-width $\epsilon$ on the output axis and a window of half-width $\delta$ on the input axis appears whose graph stays inside the band; here $\epsilon =0.5$ at $a=1$ leaves $\delta =0.22$. Shrinking the band shrinks the window, and the limit exists when a window survives every shrink. Status: schematic.

Continuity is the property a loss curve requires if a small change in a parameter is to produce a small change in the loss. The loss curve is the graph of the loss as training proceeds, and a parameter is one number the model may adjust. Continuity alone is still insufficient for training. A function can be continuous at a point and have no slope there, and the absolute value is the standard example [S:22:84]. Several classifiers, methods that assign a label to an input, employ a step function, constant on each side of a threshold, the input where the label switches; it is discontinuous at that threshold and needs special treatment in code [S:72:29].

Limitation. The definition promises something about a limit, not about any step you can name, and after any finite number of steps the run is still short of 2. That shortfall is the cost of the idea, because proving a limit means bounding the distance for every step past some point. A tabulation cannot establish that guarantee, so ten favourable steps remain evidence rather than proof.

The formal statement. A function $f$ approaches the limit $L$ at $x_0$ when, for every $ϵ>0$, there is a $\delta >0$ such that $|f(x)-L|<ϵ$ whenever $0<|x-x_0|<\delta$ [S:22:42]. The clause $0<|x-x_0|$ removes the point itself, so $f$ need not be defined at $x_0$ [S:22:42]. The order of the two quantifiers is the content: $ϵ$ is chosen against you, and $\delta$ is your answer. A sequence uses the same shape with an index: for every $ϵ>0$ there is an integer $K$ with $|x_k-L|<ϵ$ for all $k\ge K$.

The contraction case, with its hypotheses. Let $f$ map an interval into itself, and suppose there is a number $q$ with $0\le q<1$ and $|f(x)-f(y)|\le q|x-y|$ for all $x$ and $y$ in that interval [S:82:54]. Then $f$ has exactly one fixed point $x^{\ast }$, and the iteration $x_{k+1}=f(x_k)$ reaches it from any starting value [S:82:55]. The error obeys

$$|x_k-x^{\ast }|\le \frac{q^k}{1-q}|x_1-x_0|.$$

That line bounds the error at step $k$ by a geometric factor times the size of the first step.

The derivation. The contraction inequality gives $|x_{k+1}-x_k|\le q^k|x_1-x_0|$. Telescoping the gaps gives $|x_m-x_k|\le (q^k+\cdots +q^{m-1})|x_1-x_0|$, and the geometric sum is at most $q^k/(1-q)$ [S:82:57]. Let $m$ grow and the bound passes to the limit. The omitted step is completeness of the real line, which turns a Cauchy bound into an actual limit.

The edge case. The factor must be one number for the whole interval. Take $f(x)=x+1/(1+x)$ on the positive reals. For $x\ne y$,

$$|f(x)-f(y)|=|x-y|\left(1-\frac{1}{(1+x)(1+y)}\right)<|x-y|,$$

so every pair of points moves closer. Yet $f(x)-x=1/(1+x)>0$, so no fixed point exists and the iteration escapes to infinity. Pointwise shrinking is not the hypothesis. A uniform $q<1$ is. An interval open at one end fails for a different reason: $f(x)=x/2$ on $(0,1)$ shrinks every distance by the same factor $1/2$, and its only fixed point, 0, lies outside the interval [S:82:54].

What L1 hid. L1's example had $q=1/2$ and approached 2 from below, so it showed equality at every step. The theorem gives a bound, and the approach need not be monotone. L1 also fixed one starting value and one rule; the theorem allows any starting value in the interval and any map that shrinks distances. It also wrote that the run enters the tolerance band and never leaves it. That holds only from the index $K$ onward, and $K$ depends on $ϵ$.

Where the analogy breaks. The band picture invites you to choose the tolerance after watching the run. The definition fixes $ϵ$ first, then demands a $\delta$ or a $K$. Swapping that order gives a different and weaker statement, and no tabulation of early steps can supply the guarantee.

Original sources. The definition you read is Definition 2.1.2 in Trench, stated on a deleted neighbourhood so that the value at the point is excluded [S:22:42]. Trench states it for a function; the sequence case is the same clause with integers in place of the real variable. The contraction mapping theorem, with existence, uniqueness, the algorithm and the exponential rate, is Theorem 3.1 in Zhao, proved in Box 3.1 [S:82:54].

An open question. How fast optimisation converges without convexity has no general answer. For $B$-Lipschitz convex functions on a ball of radius $D$ in ${\mathbb{R}}^d$, queried through a first-order gradient oracle, Bach's construction gives a lower bound after $k<d$ iterations, and the restriction matters: methods such as the ellipsoid method then converge linearly with a constant that depends on $d$ [S:77:457][S:77:459]. For gradient methods on convex problems, the upper bounds of his Chapter 5 and the lower bounds of §15.2 match up to constants [S:77:457]. The nonconvex case is not covered by that section: its analyses bound the gradient rather than the suboptimality, and no matching pair is stated there. What is unsettled is which structural assumptions on a real training loss give a useful rate. Evidence that would settle a case: matching upper and lower bounds on a function class that contains the trained model, checked against measured iteration counts on runs the bound was not fitted to.

A frontier result. Value iteration for a Markov decision process has the form $v_{k+1}=f(v_k)$ for the Bellman optimality operator. With a discount factor between zero and one and bounded rewards, that operator is a contraction with modulus the discount factor in the maximum norm, so the theorem above gives existence, uniqueness and a geometric rate for the value function [S:82:57]. The same bound that controls $x_k$ controls the estimate of value. A smaller discount factor tightens the bound, and the bound is available before the first iteration runs.

What to read next. Read Trench §2.1 for the definition and its uniqueness proof [S:22:42]. Read Zhao §3.3 for the contraction proof and the Bellman application [S:82:54]. Read Chapter 15 of Bach for lower bounds and for what a worst case does and does not say [S:77:457]. Keep the two definitions side by side; the only change is the object that carries the index.