The bootstrap

Five reaction times, measured in hundredths of a second, read 2, 4, 4, 6 and 9. Their average is 5. If you ran the experiment again with five new volunteers, would the average still be 5?

The honest answer is that five numbers cannot tell you, and the textbook formula for an error bar assumes the data follow a particular shape — for example, a bell-shaped curve — that you may not believe. The section question is how to get an error bar from one sample, and the answer is to treat the sample as a stand-in for the population and to draw new samples from it. That procedure is the bootstrap, and it turns one dataset into a thousand plausible datasets without collecting a single new measurement. !The bootstrap for the mean of twenty task times (data/c15.s04.L1.json, seed 20261108, 2000 resamples). The rug along the bottom is the twenty observed times, from 4.6 to 56.7 minutes, whose mean is 22.36. The bars are the means of the 2000 resamples. Their spread is the error bar, and the middle 95% of them runs from 16.08 to 29.35 minutes. No new measurement was taken: the sample stands in for the population, and the interval says how far the mean would move if the experiment were run again. The figure is re-simulated from the lab's fixed seed. Status: re-simulated.

The recipe is short. Draw n rows from your n rows, with replacement. In a typical resample some rows appear more than once and some do not appear at all: the expected number of distinct rows is roughly six in ten, but a particular resample can include every row exactly once [S:64:270] [S:50:222].

Compute the statistic — the number you care about, such as a mean or a median — on the new sample, which is called a resample. The value it produces is a replicate. Repeat this B times, with B large, and record the statistic each time [S:50:222].

The spread of those B values estimates the standard error of the statistic, which is the standard deviation of its value across repeated samples from the population — a thought experiment, since you cannot collect them — [S:50:222]. A wide spread means the estimate would move a lot if the data were collected again, and a narrow spread means it would not.

The bootstrap replaces a formula with a simulation, and it buys generality at a price. Formula-based standard errors are fast and reliable when their assumptions hold, and statistical software supplies them for a straight-line fit, meaning a linear regression [S:50:220]. The bootstrap applies to many statistics, including some whose standard error has no closed form, provided the statistic is a smooth enough function of the sample and the sample is a fair stand-in for the population [S:50:220]. It is not universal. The price is computation, plus that dependence on the sample.

The count B is your choice, and the estimate of the spread settles as B grows [S:50:222]. The arithmetic behind the spread is the arithmetic of an ordinary standard deviation. Average the squared distances from the mean of the replicates, divide by B − 1 rather than B because the mean is itself estimated from the replicates, and take the square root [S:50:222]. Nothing in that calculation knows what the statistic means, which is why the method covers so many of them.

Worked example. Start from the five values 2, 4, 4, 6 and 9, whose mean is 5. One resample draws 2, 2, 4, 4 and 6, and its mean is below.

$${\bar{x}}^{\ast }=\frac{2+2+4+4+6}{5}=\mathrm{3.6.}$$

The star marks a value computed on a resample, and the bar marks an average.

A second resample draws 4, 6, 6, 9 and 9, and its mean is 34/5, or 6.8. The first replicate sits 1.4 below the original mean of 5, and the second sits 1.8 above it. A third resample might draw 2, 4, 6, 6 and 9, whose mean is 27/5, or 5.4, and a fourth might repeat a single value several times. None of the replicates is a new experiment, and together they show the kind of movement that a new experiment would produce.

With a thousand replicates you would have a thousand means, and their spread is the error bar. Report the standard deviation of the replicates, or report a bootstrap interval, a range that holds most of the replicates, and say which one you are reporting. The number is an estimate of how much the mean would move, not a statement about any one volunteer.

Think of a photograph of a crowd, and imagine building new crowds by copying people out of the photograph at random. The new crowds vary in roughly the way a genuinely new crowd would vary. The comparison breaks because copying cannot produce a person who was not in the photograph, so a rare type that your sample missed stays missing from every replicate.

The median makes the point concrete, and it is the example that shows why the method travels. The median of the five values is 4, the median of the first resample is also 4, and the median of the second is 6. Nobody needs a formula for the median's standard error, because the replicates supply the spread directly [S:50:234]. The same trick works for a tenth percentile, the value below which a tenth of the population falls, because the replicates supply its spread the same way [S:50:234]. It does not work for a maximum: every replicate is capped by the largest value you saw, so the spread cannot represent the chance that a new sample holds a larger one.

It helps to keep three levels apart, because the whole method rests on the distinction. The population is the set of all the measurements you did not take, the sample is what you hold, and the replicates are copies of the sample drawn by the computer. The bootstrap estimates the spread across samples by looking at the spread across replicates, and the estimate is only as good as the resemblance between the first two levels.

The method has real limits, and the limits follow from the recipe. The replicates can only reuse what the sample holds, so a value that never appeared cannot appear in any of them. With five rows the replicates are coarse, and with dependent rows, such as visits from the same patient, drawing one row at a time breaks the dependence and understates the spread. The bootstrap also measures variability rather than bias: if the sample is systematically unrepresentative, every replicate inherits the same fault.

The formal statement. Let the rows be an independent sample from an unknown distribution, and let the statistic be any function of the sample. The bootstrap replaces the unknown distribution by the empirical one, which puts equal weight on each observed row, and draws new samples of the same size from it with replacement [S:64:268]. For the mean, write ${\bar{x}}_b^{\ast }$ for the value on replicate $b$ and ${\bar{x}}^{\ast }$ for the average of the replicates. The error bar is

$$\widehat{\mathrm{SE}}=\sqrt{\frac{1}{B-1}\sum _{b=1}^B{\left({\bar{x}}_b^{\ast }-{\bar{x}}^{\ast }\right)}^2}.$$

The line is the ordinary standard deviation of the replicate values around their own average, so it measures spread and not bias [S:64:268]. The same recipe covers any other statistic, with that statistic in place of the mean.

The derivation. The argument is the plug-in principle and it has one step. The sampling distribution of a statistic is a function of the distribution the rows come from. You do not know that distribution, so you substitute the closest one you hold, the empirical distribution. Its sampling distribution is computable by simulation, and drawing from it is exactly drawing rows with replacement. The step I omit is the theorem that the substitution converges. That theorem needs conditions on the statistic, and those conditions are where the method fails.

The edge case. Take the maximum of the sample. Every replicate is built from observed values, so every replicate maximum is at most the largest observed value, and the replicate spread cannot represent the chance that a new sample holds a larger one. L1 says the method cannot invent values. The maximum shows that the missing values are exactly the ones that set the error bar. A second failure appears when you use the bootstrap for prediction error: fitting on a replicate and scoring on the original rows gives an optimistic number, because the two sets share rows [S:64:269]. The leave-one-out bootstrap and the .632 estimator repair that overlap, at the cost of a bias of their own [S:64:270].

What L1 hid. L1 presented the spread of replicates as an estimate of the standard error. Formally the replicates estimate the sampling distribution under the empirical distribution, while the target is the sampling distribution under the population. The two differ by the distance between those distributions, and nothing in the recipe measures that distance. L1 was safe because its sample contained the values that set the spread.

Where the analogy breaks. The photograph analogy suggests that the new crowds vary as a genuinely new crowd would. Formally, the replicates vary as draws from the photograph vary, which matches draws from the population only when the photograph is a fair sample of it. The analogy also hides the discreteness: the replicate distribution has one atom per observed row, so every interval it produces is a statement about those rows.

Original sources. ESL records that the bootstrap is due to Efron (1979), with Efron and Tibshirani (1993) as the overview, and that Efron (1983) proposed the optimism and .632 estimates of prediction error [S:64:276]. ISL gives the recipe and the variance formula that L1 follows [S:50:222]. The corpus does not contain Efron's papers, so the text cites them through this pointer and does not quote them.

An open question. The field has not settled which statistics take a valid bootstrap. The recipe assumes the empirical distribution is a fair stand-in for the population, and the guarantee is asymptotic. A maximum shows the limit: every replicate is bounded by the largest observed value, so the replicate spread cannot represent the chance of a larger one. Evidence that would settle a given case: a simulation with a known population, comparing nominal with actual coverage across sample sizes, together with a theorem that names the condition the statistic satisfies.

A frontier result. The bootstrap's advantage over a formula is that it can carry the cost of choices made by looking at the data. In ESL's spline example the smoothing choice is remade inside each replicate, so the percentiles of the fitted curves capture variability from the noise and from the selected smoothing value, which no closed-form standard error covers [S:64:286]. The same device supplies the bootstrap p-value and interval for a classifier's F-score, where no formula is available [S:96:104]. The price is that each replicate refits the whole procedure, so the cost is the cost of one search multiplied by the number of replicates.

What to read next. Read ISL §5.2 for the recipe and the standard-error formula [S:50:222], then ESL §7.11 for the prediction-error version and the reason the naive version is optimistic [S:64:269]. For the overlap correction, read the leave-one-out bootstrap and the .632 estimator in the same section [S:64:270]. For classifier evaluation, read Eisenstein's significance section [S:96:104].