The central limit theorem

A study of tablet users reports a mean age of 34 years and a standard deviation of 15 years. Take a random sample of 100 users and ask how the sample average would move from one sample to the next [S:136:412]. The answer is a spread of 1.5 years, and the shape of those averages, called the sampling distribution, is approximately normal, the symmetric bell-shaped curve [S:136:412].

Both halves of that answer are the central limit theorem at work, and the theorem is the reason two numbers describe a sample average. The theorem says that if n values are independent draws from one distribution with mean μ and finite variance σ², the standardised sum converges in distribution to the standard normal [S:19:17]. Converges here means that the probability of any range settles on the value the normal curve gives for it. The raw sum converges to nothing: its mean is n times μ and its variance is n times σ², so its distribution keeps moving and spreading [S:41:241]. !Densities of the sample mean for four sources, each drawn on an axis measured in units of sigma over the square root of n. In the left column the sources look nothing alike: two spikes, six spikes, a skewed wait and two humps. By the right column the four are hard to tell apart, and their peaks all sit within 0.02 of 0.399, the peak of the standard normal drawn behind them. The source shape does not survive; its centre and its spread do. Status: re-simulated.

Variances of independent values add, which is where the n times σ² comes from. Divide the sum by n and the same statement describes the average, whose mean is μ, whose spread is σ divided by the square root of n, and whose shape is the same bell.

The bell keeps appearing because sums of many small contributions are everywhere in measurement. A measurement is often the sum of many small influences, some genetic and some environmental, and a sum of many small effects lands near a normal curve [S:41:242]. That is why one shape turns up in heights, in measurement errors and in the average of a hundred dice, with no bell built into any single cause.

That sentence says the sum's spread grows with the sample, but more slowly than the sum itself grows, so the spread of the average shrinks as the sample gets larger.

Worked example. A fair die has mean 3.5, because each of the six faces is equally likely. Its variance is 2.92, the average squared deviation of the faces from 3.5, so its standard deviation is 1.71, and those two numbers are all the theorem needs from the die.

Now roll 100 of these dice and add the results together. The sum has mean 350, because 100 times 3.5 is 350, and it has spread 17.1, because the square root of 100 is 10 and 10 times 1.71 is 17.1, a product you can check on any calculator.

Divide by 100 and the average has mean 3.5 and a spread of 0.171, small enough that one sample of 100 rolls almost always lands close to the centre. Read that spread as a probability: about two thirds of samples land within one spread of 3.5, and about nineteen in twenty within two.

Read those two spreads side by side: the sum wanders over tens of points, while the average stays within a small fraction of a point of 3.5. The sum's spread grew by a factor of ten, and then dividing by the sample size cut it by a hundred, which is why an average over many rolls is a far better estimate of 3.5 than any single roll is.

The shape claim is separate from the spread claim, and it is the part that earns the theorem its name, because the shape is what tells you the probability of any range. Once you know that the average is approximately normal, its centre and its spread determine everything you might ask about it. The probability that the average lands in a range becomes an area under one curve: the area between two values is the chance of landing between them [S:135:327]. You never need the distribution of a single die, a single user or a single error.

That convenience explains why the bell curve is the default description of uncertainty, and it also marks the limit of the default. A bell is earned when the quantity is a sum of many small and independent contributions, and it is not earned when one contribution dominates the rest, which is the case the next section takes up.

The theorem costs something, and the cost hides in the word approximately. The gap between the standardised sum and the normal curve is at most ρ/(σ³√n), where ρ is the average absolute cube of one deviation [S:19:22]. That order cannot be improved in general: for a fair coin the chance of landing exactly on the mean is of order one over the square root of n, while the normal curve gives zero there. A symmetric source can do better than the bound, so how sharp the sketch is depends on the source as well as on n.

Skewed sources, sources whose one side stretches further than the other, need more values before the sketch becomes usable [S:41:242]. Dependent values break the argument entirely. The far tails, the rare values a long way from the centre, are where approximations of this kind are weakest. It describes the middle of the average with some accuracy, and it says nothing useful about the size of a rare surprise, which is the subject of the next section.

The formal statement. Let the draws be independent, from one distribution with mean m and finite variance σ². Write S for the sum of the first n. The Lindeberg–Lévy theorem says that S minus n m, divided by σ times the square root of n, converges in distribution to the standard normal [S:19:17]. Convergence in distribution means that the cumulative distribution at every point converges. The three hypotheses are independence, one common distribution, and finite variance.

The derivation, and the step omitted. The standard proof works with the characteristic function, which is the Fourier transform of the distribution. Independence makes the characteristic function of the sum the product of the n factors. A Taylor expansion to second order turns that product into the characteristic function of the normal curve in the limit. The omitted step is Lévy's continuity theorem, which says that convergence of characteristic functions implies convergence of distributions. That theorem is what allows the algebra to speak about distributions at all, and its proof is where the measure theory lives.

The quantitative version. The word approximately has a rate. The Berry–Esseen theorem bounds the gap between the distribution of the standardised sum and the normal by a ratio of the third absolute moment to σ cubed, divided by the square root of n [S:19:22]. Two consequences follow. The gap shrinks slowly. It also cannot be removed in general: for a fair coin the exact probability of landing on the mean is of order one over the square root of n, while the normal curve gives zero there [S:19:22].

The edge case. The theorem needs a finite variance, and the core's "almost any distribution" is the source's shorthand rather than a licence. A Pareto distribution with a tail index below two has no finite variance, and its standardised sums do not approach a bell [S:41:243]. The core said that skewed sources need more values before the sketch becomes usable. The sharper statement is that an infinite variance puts the source outside the theorem entirely, and no sample size repairs that.

What the core hid. The limit is a statement about the standardised sum, not about the raw sum. The raw sum has mean n m and variance n σ², and its distribution spreads out; it does not converge to anything. Subtracting n m moves the centre to zero, dividing by the square root of n fixes the width, and only then does a shape appear.

Where the analogy breaks. "A sum of many small contributions" describes when the bell turns up in nature, not what the theorem says. The theorem is about any source with a finite variance, additive or not. The story about genes and environment is a plausible account of why the hypothesis holds for a measurement, and it is not a proof.

Original sources. The theorem is Theorem 1.3.2, the Lindeberg–Lévy central limit theorem, in Vershynin [S:19:17]. The coin case carries its own name, the de Moivre–Laplace theorem [S:19:18]. The quantitative version is the Berry–Esseen theorem, stated in the same chapter with the error of order one over the square root of n [S:19:22]. Downey gives the conditions in plain words and shows what happens when they fail [S:41:242].

An open question. The theorem is asymptotic, and the question a practitioner faces is how large n must be before a normal interval has the coverage it advertises. There is no general answer, because the rate depends on the skewness of the source [S:41:242]. Evidence that would settle a case: nominal against actual coverage, measured across several sample sizes and several named distributions, on data where the true answer is known. Until that curve is drawn for your data, the honest statement is that the bell is a sketch whose sharpness you have not measured.

A frontier result. The slowness of the approximation is the reason concentration inequalities exist. Berry–Esseen puts the error at order one over the square root of n. Vershynin points out that this error is too large to prove the exponential tail of the sum, so the exponential bound has to be derived by a different route [S:19:22]. Section 12.5 takes that route. The central limit theorem describes the middle of the average; the tails need a tool that does not pass through it.

What to read next. Read section 1.3 of Vershynin for the theorem [S:19:17] and section 2.1 for the failure of the approximation [S:19:22]. Read section 14.4 of Downey for the conditions and the pictures [S:41:242]. Then read section 12.5 of this book, which uses the gap.