What an estimator is

You weigh the same parcel five times and write down 2, 4, 6, 8 and 10 grams. The scale is noisy, so the readings disagree. You need one number, and two rules compete: report the first reading, which gives 2, or report the average, which gives 6. One rule is the better guess. The chapter asks what the best guess is and how sure we can be; this section takes the first half of that question.

A rule that turns data into a guess is an estimator. More exactly, an estimator is a procedure applied to data that returns a guess about a fixed but unknown quantity [S:63:189]. The number it returns on the data you hold is the estimate, written $\hat{\theta }$, where the Greek letter theta stands for the unknown quantity itself, and the rule stays fixed while the data change. That distinction is the whole subject, because a rule can be sensible even when one of its outputs is poor. The bracketed code marks the page of the source, and you can skip it on a first read. !Three rules read the same five-point population (2, 4, 6, 8 and 10 grams, each equally likely), whose true average is 6. Reporting the first reading repeats the population exactly, five values each with chance 1/5 and an average squared error of 8. Averaging two readings gives the nine values 2 to 10 in steps of 1, most often 6, and halves the error to 4. Averaging five independent readings gives 21 values in steps of 0.4, still centred on 6 but with most of its chance between 4 and 8; its average squared error is 8/5 = 1.6. All three rules are unbiased, so the guesses differ only in how far they scatter, and each further average narrows the distribution. The heights share one probability scale. Status: synthetic.

Suppose you could repeat the experiment: five new readings, then another five, and so on. Each repetition of the whole experiment is a draw, and each draw produces a different estimate.

The collection of all those estimates is the estimator's sampling distribution [S:63:185]. The data are random, the true weight is a fixed unknown number, and the estimate is therefore a random quantity [S:63:184]. Uncertainty here means how much the guess would move if the data moved [S:63:184]. This is the frequentist view, which treats uncertainty as the variability of estimates over repeated draws, and it is the one this chapter uses.

Now compare the rules on a population small enough to enumerate. Let the five possible readings be 2, 4, 6, 8 and 10, each equally likely, so that the true average is 6. The population variance is the average squared distance of a reading from 6: (16 + 4 + 0 + 4 + 16)/5 = 8.

Rule F reports the first reading. Rule A reports the average of two readings. List all 25 ordered pairs of readings, and score each rule by its squared error against 6, the square of how far its guess lands from the truth.

The first row is easy to check. The squared distances of the five readings from 6 are 16, 4, 0, 4 and 16, and their average is 8. The second row uses a fact about averages. Two draws are independent when one tells you nothing about the other. Independent errors do not systematically reinforce each other, so averaging two readings halves the variance and the average squared error falls from 8 to 4.

Neither rule is systematically high or low. When the expected difference from the truth, the average of estimate minus truth over repeated draws, is zero, the estimator is unbiased [S:63:190]. Unbiasedness alone does not make a rule good. Looking only at the first reading is unbiased, yet it lands further from the truth than the average on most samples [S:63:190-191]. Rule A wins because its guesses scatter less, not because it aims better.

Rule A used two readings, and more readings help. The average of all five has average squared error 8/5 = 1.6, because averaging five independent draws divides the variance by five. To halve the average squared error you need twice the readings; to halve the typical error, the square root of that average, you need four times. That is a square-root rate: the error falls like one over the square root of the number of readings, and Section 13.5 makes the rate precise.

The same calculation gives the whole sampling distribution, not only its average. The average of all five readings is not a spike: it takes 21 values from 2 to 10, centred on 6, with variance 8/5 = 1.6. The first reading takes five values. A distribution concentrated on a single value would need a population whose readings never vary. Real estimators sit between these extremes, and their distributions are what the next sections summarize with two numbers.

The point is the distribution, not the average. "Best" is not a property of a single number. It is a property of the rule's whole distribution of guesses, and it depends on the cost you assign to being wrong, the penalty you pay for a miss. Squared error is the cost used here because it decomposes cleanly, as the next section shows. A rule that is usually close and occasionally wild can lose to a rule that is always slightly off.

The average is not always the right rule. If one reading is a transcription error, the average drags it into the answer, while the middle reading ignores it. The middle reading is the median, and it is biased for some populations and steadier for others. Which rule wins depends on the population and on the cost, which is why the next two sections separate the sources of error instead of crowning one rule.

The same rule can win on one population and lose on another. If the five possible readings were 0, 0, 0, 0 and 30, the true average would still be 6, so the average rule would land 6 grams high every time while the median would stay at 0. Neither rule is universally best. You choose a rule after naming the population, the cost and the data you can afford to collect.

The idea costs something. To compute a sampling distribution you need the process that generated the data, or a model of it: a written description of how the readings arise, with the noise included. You rarely have either. With one dataset you can estimate the spread but never see it directly. The criterion also matters, because average squared error punishes a rare large miss heavily, and that may be the wrong cost when one bad guess ruins the day.

The formal statement. Fix a family of distributions $p_{\theta }$ on a sample space, indexed by an unknown $\theta \in \Theta$. An estimator is a measurable function $T$ that maps a dataset $D=\{x_1,\dots ,x_n\}$ to a value in $\Theta$ [S:63:184]. Its risk at $\theta$ is $R(\theta ,T)=E_{\theta }[L(\theta ,T(D))]$, where the expectation runs over datasets drawn from $p_{\theta }$. With squared loss the risk is the mean squared error. The rule $T$ is unbiased when $E_{\theta }[T(D)]=\theta$ for every $\theta$ [S:63:190].

The derivation. L1's table follows from two lines of algebra. For independent draws with variance ${\sigma }^2$, the average of $n$ draws has variance ${\sigma }^2/n$. L1's population has ${\sigma }^2=8$, so the mean of two draws has variance 4 and the mean of five has variance 1.6. Both means are unbiased, so each mean squared error equals its variance. The first reading has variance 8 and bias zero. Risk ranks the rules, and the average wins on this population at every $n>1$.

The edge case. The ranking is not universal. Take draws from a Cauchy distribution. The sample mean of $n$ draws has the same Cauchy distribution as one draw, so averaging buys nothing and the mean has no finite variance. The sample median stays consistent and has finite spread. A reader who carries "more data always helps" from the five-reading example predicts the opposite here. L1's example works because the population has a finite variance, and L1 never states that hypothesis.

What L1 hid. L1 treated "best" as one ranking. The formal object is a risk function $R(\theta ,T)$, one number per candidate truth. Two rules can cross: each wins on part of $\Theta$. L1's table is safe because it fixes one population and one loss. The table also hides that $T$ is a function of the whole dataset, so its distribution depends on $n$ and on the sampling scheme, not on the numbers in the table alone.

Where the analogy breaks. A scale has a fixed offset. An estimator's bias is a function of $\theta$ and of $n$, so the same rule can be unbiased at one truth and biased at another. The parcel story also suggests that one reading is a fixed error. In fact the error is a random variable, and you need its distribution to compare rules. The scale analogy is safe only when the offset is constant and the loss is squared error.

Original sources. Murphy states the frequentist frame that L1 uses: the data are random, the parameter is fixed, and uncertainty is the distribution of the estimate across repeated datasets [S:63:184]. He calls an estimator a decision procedure that maps data to an action [S:63:184]. The sampling distribution is the distribution that this procedure induces [S:63:185]. Murphy states the frame without naming an earlier origin for the sampling-distribution idea, so the text should not claim one.

An open question. Conformal prediction builds a set $C(X_{\text{test}})$ with $\mathrm{Pr}(Y_{\text{test}}\in C(X_{\text{test}}))\ge 1-\alpha$ when the calibration pairs and the test pair are i.i.d. [S:48:6], and the theorem also holds under the weaker condition of exchangeability [S:48:50]. It assumes nothing about the model or the parametric form of the distribution, and the guarantee is marginal: it averages the probability over calibration and test draws [S:48:4]. It does not hold for every subgroup, and adaptive sets tend to undercover hard subgroups and overcover easy ones [S:48:6]. In the most general case, distribution-free conditional coverage is impossible rather than unresolved: marginal 90 percent coverage can coexist with group coverages of 5, 14, 100 and 80 percent [S:48:13]. What remains open is how close the practical relaxations, such as feature-stratified and class-conditional coverage, can come at finite $n$. Evidence that would settle a given case: a procedure with a finite-sample lower bound on coverage for every subgroup, checked on a benchmark whose hard subgroups are fixed before the analysis.

A frontier result. Split conformal prediction turns any heuristic score into a prediction set with the guarantee that the truth is included with probability at least $1-\alpha$, provided the calibration pairs and the test pair are exchangeable [S:48:6] [S:48:50]. The method depends on the idea in this section. Its guarantee is a statement about repeated draws of calibration and test data, which is the sampling-distribution view applied to a set-valued estimator.

What to read next. Read Murphy 4.7.1 for the definition and 4.7.2 for the Gaussian approximation of the sampling distribution [S:63:184] [S:63:185]. Then read Angelopoulos and Bates for the distribution-free alternative, its coverage theorem, and the limits of conditional coverage [S:48:4] [S:48:6] [S:48:13].