Consistency and asymptotics

A coin with head probability 0.5 is tossed 10, 100 and 10,000 times. The standard error of the estimated probability is 0.158, 0.05 and 0.005. A tenfold increase in the sample divides the error by a little more than three; a hundredfold increase divides it by ten. The estimate does not become correct by magic; it becomes correct because the data accumulate, and the rate at which that happens is the subject of this section.

An estimator is consistent when its estimate converges to the true value as the sample grows [S:66:120]. The sample mean of a coin is the fraction of tosses that land heads, and it is consistent for the head probability, as is the maximum likelihood estimate. Consistency is a statement about a limit. It says where the rule ends up, not how fast it gets there, and it says nothing about the estimate you have at n = 20. !The standard error of a coin estimate at p = 0.5, the square root of 0.25/n, on log-log axes. The relation is a straight line of slope minus one half, so every factor of four in n buys a factor of two in precision: 0.158 at ten tosses, 0.10 at twenty-five, 0.05 at a hundred, 0.01 at 2,500 and 0.005 at ten thousand. Reading the line backwards gives the sample size an experiment needs, and the marked point at 2,500 tosses is what a standard error of 0.01 costs. Consistency is the statement that the line keeps falling; the slope is what makes the promise expensive. Status: synthetic.

Convergence here is a statement about probability, not about a single run. At any fixed sample size there is a chance that the estimate is far from the truth, and that chance shrinks as n grows. The guarantee is that the chance tends to zero, not that any one experiment lands close. This is why a consistent rule can still embarrass you on a small sample. !The exact sampling distribution of the estimated head probability, with the Gaussian the asymptotic result promises drawn over it. At ten tosses and p = 0.5 the bars are visibly uneven and the curve misses them in the tails. At a hundred tosses the bars and the curve agree, which is what lets an estimate plus or minus two standard errors stand in for an interval. The third panel keeps a hundred tosses and moves p to 0.01: the estimate piles up against zero, the curve reaches past zero to negative estimates, and the same formula that worked beside it is now optimistic. The approximation is a limit approached at a rate that depends on where in the range the parameter sits. Vertical scales differ between panels; the bars are binomial mass per unit of p-hat. Status: synthetic.

Two consistent rules need not be equally precise. For Gaussian data the variance rule that divides by n has mean squared error ${\sigma }^4(2n-1)/n^2$, while the unbiased rule that divides by n−1 has $2{\sigma }^4/(n-1)$ [S:135:594]. At ten observations the biased rule's error is about six sevenths of the unbiased rule's, and the ratio approaches one as n grows. That ratio is statistical efficiency: one consistent estimator reaches a fixed error with fewer examples than another, and the comparison is made at a fixed sample size [S:66:121].

Efficiency is the constant in front of the rate, not the rate itself. Regular estimators share the $n^{-1/2}$ rate, so what separates them is the size of the error at a given n, and the Cramér–Rao bound $1/(nF)$ is the best possible constant [S:35:381]. It is measured against rivals and against that bound. This is why the asymptotic story and the finite-sample story must be told together.

The sampling distribution also settles into a shape. As the sample grows, the distribution of the maximum likelihood estimate becomes Gaussian, centred on the true value, with a variance set by the Fisher information [S:63:185]. This is asymptotic normality. It is why the bell curve appears in every discussion of standard errors, and why intervals built from it work in large samples. The estimate is then asymptotically efficient: it reaches the Cramér–Rao bound in the limit, under regularity conditions, the smoothness assumptions that let derivatives pass under the integral [S:66:121].

The approach is gradual. With ten observations the sampling distribution of a coin estimate is visibly asymmetric and far from Gaussian; with a hundred it looks Gaussian [S:63:187]. The approximation is not a switch that flips at some sample size. It improves as n grows, and the quality of the improvement depends on the model and on the value of the parameter.

The bell curve is a limit, and limits are patient. At n = 100 the approximation is decent for a coin near 0.5 and poor near 0.01, where the estimate piles up against zero. The same formula can be optimistic in one regime and conservative in another. Reading the asymptotic result without the regime is how intervals acquire false confidence.

Return to the numbers. At $p$ = 0.5 the standard error is the square root of 0.25/$n$. At n = 25 it is 0.1. At n = 100 it is 0.05.

Quadrupling the sample halves the error, and the same factor of four buys the same halving at every scale. The pattern is the square-root rate from the previous section, now read as a statement about what happens as n grows without bound.

The three sample sizes in the opening make the promise concrete. At n = 10 the estimate is too noisy to distinguish a fair coin from a 60 percent coin. At n = 100 the standard error is 0.05, so that distinction becomes visible. At n = 10,000 the estimate is precise enough to detect a two-point difference. The asymptotic statement is the same in all three cases; only the finite sample tells you what you can see.

Asymptotic normality buys something concrete. It turns the standard error into an interval. The estimate plus or minus about two standard errors covers the truth in roughly 95 percent of repeated samples, because a Gaussian puts 95 percent of its mass within $1.96$ standard deviations of its mean. The interval is approximate, and its coverage degrades when n is small or p is near an end. But it gives a usable answer to the chapter's second question, how sure we are, without requiring a simulation of the sampling distribution.

Asymptotics can mislead. A rule that is consistent under independent observations can fail when the data are dependent, when the model is wrong, or when the distribution has heavy tails. Heavy tails mean extreme values arrive often enough that the variance or the bell curve stops being a useful description. A consistent estimator can be worse than a biased one at the sample size you actually have. And convergence in probability is not a guarantee for any single run, which is why the next section keeps the finite-sample view.

The formal statement. An estimator $T_n$ is weakly consistent for $\theta$ when, for every $ϵ>0$, the probability $P_{\theta }(|T_n-\theta |>ϵ)$ tends to zero as $n$ grows. It is strongly consistent when $T_n$ tends to $\theta$ almost surely [S:66:117]. Asymptotic normality is a second statement. If the parameter is identifiable and the model is regular, then $\sqrt{n}(T_n^{\mathrm{MLE}}-\theta )$ converges in distribution to a Gaussian with mean zero and variance $1/F(\theta )$ [S:63:185]. The two statements answer different questions. Consistency says where the estimate goes; asymptotic normality says how the standardized error is distributed on the way.

The derivation. Consistency of the sample mean follows from Chebyshev's inequality. For independent draws with finite variance ${\sigma }^2$, $P(|T_n-\theta |>ϵ)\le {\sigma }^2/(n{ϵ}^2)$, and the right side tends to zero. For the sample mean, asymptotic normality follows from the central limit theorem: $\sqrt{n}(T_n-\theta )/\sigma$ converges to a standard Gaussian, so $\sqrt{n}(T_n-\theta )$ converges to a Gaussian with variance ${\sigma }^2$. For the Bernoulli mean, ${\sigma }^2=p(1-p)$ and $F(p)=1/(p(1-p))$, so ${\sigma }^2=1/F(p)$ and the two statements agree. The general proof for the maximum likelihood estimate expands the score around the truth and needs uniformity in $\theta$; I omit that step and point to the theorem [S:63:185].

The edge case. The first reading is unbiased and inconsistent, because its distribution does not change with $n$ [S:66:117]. Consistency is not implied by unbiasedness, and the two properties come apart here. A second edge: the Gaussian limit needs an identifiable parameter. When several parameter values give the same distribution, the theorem does not apply, and the estimate need not settle on a bell curve [S:63:185].

What L1 hid. L1 said convergence concerns probability rather than one run. The formal statement adds that the mode of convergence matters: almost sure convergence is stronger than convergence in probability. L1 also hid the standardization. Asymptotic normality describes $\sqrt{n}$ times the error, not the estimate, so the spread of the estimate still shrinks like one over the square root of $n$.

Where the analogy breaks. "The estimate becomes correct" suggests a monotone approach. A consistent sequence can move away from the truth at a finite $n$ and return later. The guarantee is about the tail of the sequence, not about the path, and no finite sample is promised anything.

Original sources. The notion of consistency of estimators dates back to Fisher (1922) [S:79:97]. Goodfellow gives the formal definition, defines convergence in probability, and separates weak from strong consistency [S:66:117]. Murphy states the asymptotic normality of the maximum likelihood estimate and gives the Gaussian limit with variance set by the Fisher information [S:63:185]. L1's two limits are these two statements.

An open question. The Gaussian limit assumes an identifiable parameter [S:63:185]. In an overparameterized model many parameter values produce the same distribution, the theorem does not apply, and the limit distribution of the estimate is not the Gaussian in the statement. What replaces it is not settled in general. Evidence that would settle a given model: a limit theorem for that model, with the predicted distribution checked against repeated fits at growing sample sizes.

A frontier result. The bootstrap approximates the sampling distribution by resampling the observed data, which is useful when no closed form is available [S:63:186]. It is the practical form of the idea in this section. You replace the unknown data distribution with the empirical one, then push the estimator through many resampled datasets. At small samples the approximation can be visibly asymmetric, and it improves as the sample grows [S:63:187].

What to read next. Read Goodfellow 5.4.5 and 5.5.2 for consistency, the two conditions that make maximum likelihood consistent, and efficiency [S:66:117] [S:66:120] [S:66:121]. Read Murphy 4.7.2 and 4.7.3 for asymptotic normality and the bootstrap [S:63:185] [S:63:186] [S:63:187]. Read Shalev-Shwartz and Ben-David chapter 7 for the historical note and a learning-theory treatment of consistency [S:79:97].