How much can one observation tell you?

You toss a coin once and see heads. The maximum likelihood estimate is 1, and a standard 95 percent confidence interval collapses to the single point 1. The interval is $\hat{p}\pm 1.96\sqrt{\hat{p}(1-\hat{p})/n}$, the usual Gaussian formula, and at $\hat{p}=1$ the quantity under the square root is zero [S:63:189-190]. The stated percentage is a property of the rule, not of this one interval: a 95 percent rule covers the truth in 95 percent of repeated samples. The estimate says the coin always lands heads and the interval says the same, so both are absurd, and the reason is not a bug in the formula.

The missing quantity is the Fisher information, which measures how sharply the likelihood changes as the parameter moves [S:63:185]. If a small change in p makes the observed data much less likely, the data pin p down well. If the likelihood is flat, many values explain the data equally well, and one observation cannot separate them. Fisher information turns that picture into a number. !One Bernoulli observation. In (a) the information 1/(p(1-p)) is 4 at p = 0.5 and about 11.1 at p = 0.9, and it grows without bound as p approaches 0 or 1: near an end, a small move in p makes the observed result far less likely. The vertical axis is clipped because the curve diverges. In (b) the standard error of the estimate is the reciprocal square root of the total information, so it is largest at p = 0.5 and falls with n: 0.5 from one toss, 0.05 from a hundred, 0.005 from ten thousand. One toss leaves a very wide interval: the information is 4, so the standard error is 0.5, and the Wald interval collapses only when the estimate sits at 0 or 1. Status: synthetic.

For one Bernoulli observation, meaning one toss coded 1 for heads and 0 for tails, the information about p is smallest at a fair coin, where it is 4, and grows without bound as p approaches 0 or 1. At $p$ = 0.9 one toss carries about 11.1. The units are inverse variance, which is why information adds when observations are independent [S:35:380]. Ten tosses carry ten times the information of one, and the estimate's spread shrinks accordingly.

The standard-error is the square root of the variance of the estimator's sampling distribution [S:63:186]. Write the information for one observation as $F(p)$ and the standard error as $\mathrm{se}$. One observation carries $F(p)=1/(p(1-p))$, and $n$ independent observations carry $nF(p)$, so the standard error is the square root of the reciprocal of that total:

The first expression says a coin near 0 or 1 is the most informative case and a fair coin the least. The second says the standard error shrinks like one over the square root of n.

Put p = 0.5. One observation carries F = 4, and the standard error of the estimate is 0.5, so one standard error on each side reaches from 0 to 1. The 95 percent band is wider still and runs outside the range from 0 to 1. One toss tells you almost nothing because the band covers everything, not because it covers half of it.

With 100 tosses the standard error is 0.05, ten times smaller. With 10,000 tosses it is 0.005. The estimate itself stays near 0.5, but the uncertainty around it contracts.

The square-root rate is the most useful number in this section. To halve a standard error you need four times the data, and to divide it by ten you need a hundred times the data. The $n^{-1/2}$ rate is shared by all regular estimators, but the constant in front is not: the Cramér–Rao bound is $1/(nF)$, and an inefficient unbiased rule has a larger variance at the same $n$. The rate is a property of the problem; the price paid at a given $n$ is a property of the problem and of the estimator.

The rate assumes independent observations. If repeated measurements are correlated, ten readings can carry little more information than one, and the standard error stops shrinking at the rate the formula promises. Dependence is the most common reason a real dataset gives less precision than its size suggests.

A lower bound makes the contraction unavoidable. The Cramér–Rao bound says that no unbiased estimator can have variance below the inverse of the total Fisher information [S:35:381]. The maximum likelihood estimate reaches that bound in large samples, which is what efficient means, and it converges to the truth, which is what consistent means [S:35:381]. You cannot buy more precision from the same data by choosing a cleverer unbiased rule.

The information is also about a parameter in a model, not about the world. A parameter can be precisely estimated in a model that is wrong, and the precision then measures the wrong thing. This is why an interval can be narrow and still miss the truth: the bound covers sampling variability, not model error.

The failed interval from the opening is instructive. The Wald interval is built from the estimated variance $\hat{p}(1-\hat{p})/n$, and that estimate is exactly zero when the estimate sits at 0 or 1, so the interval has zero width there. This is a boundary failure of the Gaussian approximation, not a broken symmetry. A different interval does repair the degeneracy at one observation: one that does not start from $\hat{p}(1-\hat{p})$, such as a Bayesian credible interval with a Jeffreys prior, stays non-degenerate [S:63:189]. Every interval is still wide, because one observation carries little information, and that is a separate fact from the collapse.

Design is a different question, and it belongs to models with inputs. A coin has no setting to choose: p is the parameter being estimated, not something you control. If you can choose where to measure, the choice acts on the inputs of a model, and the best place depends on which parameter you want to know about. What the coin illustrates is information, not design.

The bound has conditions. It applies to unbiased estimators, to smooth models, and to a stated parameterization; change the parameterization and the Fisher information changes [S:35:379-380]. It is a local statement about the truth, not a guarantee that your estimate is near it. And information about a parameter is not information about a prediction: a model can pin down its parameter and still predict badly, because the prediction depends on inputs you have not seen.

The formal statement. For a scalar parameter and a density $p_{\theta }$, the score is $s(\theta )=\partial \log p_{\theta }(x)/\partial \theta$. The Fisher information is

The two expressions agree when derivatives pass through the expectation [S:35:379]. For $n$ independent observations the information is $nF(\theta )$. The Cramér–Rao bound says that any unbiased estimator $T$ has $\mathrm{Var}(T)\ge 1/(nF(\theta ))$, under the same regularity conditions [S:35:381].

The derivation. For one Bernoulli draw, $s(p)=x/p-(1-x)/(1-p)$. Take expectations using $E[X]=p$: the score has mean zero. Put the score over the common denominator to get $s(p)=(X-p)/(p(1-p))$. Square it and take the expectation: $F(p)=p(1-p)/(p^2(1-p{)}^2)=1/(p(1-p))$. The estimate $\hat{p}=X/n$ has variance $p(1-p)/n$, which equals $1/(nF(p))$. The bound is therefore attained, and no step is omitted for this case.

The edge case, and the parameterization. L1's formula is written in the probability parameterization, where information is smallest at a fair coin and grows without bound at the ends. Change the parameter to the logit, $\theta =\log (p/(1-p))$, and the same observation carries $F(\theta )=p(1-p)$ [S:35:379-380], which peaks at $p=0.5$ and vanishes at the ends, the opposite shape. Both statements are true because they answer different questions: one measures information about $p$, the other about the log-odds. L1 displays no table of information values: 4 at $p=0.5$ and 11.1 at $p=0.9$ are both values of $1/(p(1-p))$ on the probability scale and are consistent with each other. The logit-scale value at $p=0.5$ is 0.25, so 4 cannot be read on that scale either. Fisher information transforms as $F_{\varphi }=F_{\theta }/(g^{\prime }(\theta ){)}^2$, and the Cramér–Rao bound for a smooth function $\varphi (\theta )$ transforms with the Jacobian and is unchanged [S:35:381]. Which $p$ is most informative therefore has to name the parameter being estimated, and no reparameterization rescues a claim about the coin alone.

What L1 hid. L1 presented the information as a property of the coin. It is a function of the unknown parameter, evaluated at a candidate value. The bound uses the information at the truth; you can compute only the information at the estimate. The two differ, and the difference is largest near the boundary. L1's opening uses one head, where the estimate and the collapsed interval sit at 1; the source's worked example uses one tail, where both sit at 0 [S:63:189-190].

Where the analogy breaks. "One observation is one unit" suggests that units add like coins in a jar. They add only for independent observations of the same parameter. Correlated readings share information, and a change of parameter changes the number while leaving the data untouched.

Original sources. Duchi defines the Fisher information as the covariance of the score, gives the equivalent Hessian form, and proves the Cramér–Rao bound for unbiased estimators [S:35:379] [S:35:381]. Goodfellow attributes the bound to Rao (1945) and Cramér (1946) [S:66:121]. The corpus contains the derivation, not the two original papers, so L3 should not quote them.

An open question. Fisher information depends on the parameterization [S:35:379-380]. For a neural network the empirical Fisher can be singular on a flat region, so the bound gives no usable number there [S:62:318]. The field has not settled which parameterization or which curvature matrix gives honest uncertainty for overparameterized models. The difficulty is not the algebra but the choice of the parameter whose uncertainty you report. Evidence that would settle a given case: an interval with a coverage guarantee that holds across parameterizations, checked on models whose Fisher matrix is singular.

A frontier result. Natural gradient descent uses the Fisher information as a metric and steps along $F^{-1}g$ instead of along the raw gradient [S:72:239]. The same matrix also measures the curvature of the log-likelihood, so it serves as a second-order approximation to the objective. The quantity that bounds the variance of an estimator therefore also sets a search direction, which is why information geometry connects estimation and optimization.

What to read next. Read Duchi chapter 15 for the definition, the Bernoulli example and the bound [S:35:379] [S:35:381]. Read Blondel and Roulet 9.3 and 17.3 for the Fisher matrix in differentiable programming and its use in natural gradient [S:72:239]. Read Murphy 4.7.2 for the same matrix in the asymptotic normality of the maximum likelihood estimate [S:63:185].