What a p-value means

A spam filter catches nine of twelve spam emails. A coin that says spam or not with equal odds would sometimes catch nine out of twelve as well. How often would it?

Under the boring explanation, each of the twelve guesses is a fair coin, so the twelve guesses have $2^{12}=4096$ equally likely patterns. Nine or more correct covers four counts: nine, ten, eleven and twelve. Those four counts occur in 220, 66, 12 and 1 of the 4096 patterns. There are 220 ways to be right nine times, one for each choice of the three guesses that go wrong, and 66 ways to be right ten times, one for each pair that goes wrong. !Panel (a): the null distribution for twelve independent coin guesses, one bar per count of correct guesses. Nine or more correct covers the four right-hand bars, which hold 220, 66, 12 and 1 of the 4096 equally likely patterns, so p = 299/4096, about 0.073. Panel (b): that p on a number line against the two thresholds used in the text. It lies above 0.05 and below 0.10, so the same filter fails one test and passes the other, and nothing about the filter changed between them. The figure is synthetic: it is the binomial arithmetic of the section, not a measured experiment. Status: synthetic.

Those add to 299, the number of patterns at least as extreme as the filter's result. That fraction, $299/4096\approx 0.073$, is the p-value.

The boring explanation is the null hypothesis, and the count of correct guesses is the test-statistic. More generally, the p-value is the probability of a result at least as extreme as yours in the direction the test fixes, computed under the null [S:116:179]. The section question is what such a test actually tests, and the short answer is that it tests a model of no skill rather than the filter you care about. Computing the p-value is the arithmetic at the centre of a significance test; the test also needs a null, a statistic, a direction and a level chosen before you look.

You choose a significance level alpha before looking at the data, and the usual choice is 0.05 [S:116:178]. If the p-value is at most alpha you reject the null hypothesis, and otherwise you fail to reject it [S:116:179]. Rejecting is not the same as proving, and failing to reject is not the same as accepting.

The threshold has a meaning beyond convention: alpha is the largest probability of a false alarm that you are willing to accept, where a false alarm means rejecting the null when the null is true [S:116:178]. A smaller alpha makes the test harder to pass, and the cost of that strictness is that genuine effects are missed more often.

Worked example. Compare 0.073 with a level fixed before the data. At alpha equal to 0.05 the filter does not pass, because 0.073 lies above the threshold. At alpha equal to 0.10 it passes, because 0.073 lies below the threshold, and nothing about the filter changed between those two sentences.

The p-value is a fact about the data, the null, and the chosen statistic or rejection region, so a different statistic gives a different number from the same data. The threshold is a separate decision about how much evidence you demand. The arithmetic is a binomial test, the standard exact test for a count of successes out of a fixed number of tries [S:96:104]. It treats the twelve emails as independent trials, meaning that one email's result does not change the chance for another, which is a claim about the data that you should state whenever you report the result.

The most common misreading is that the p-value is the probability that the null hypothesis is true. It cannot be, because the calculation began by assuming that the null is true [S:116:179]. The number describes the data in a world where the null holds, and it says nothing about the chance that the world is that way.

Think of the p-value as the chance of seeing a result this lopsided from a fair coin. The comparison breaks because a coin is a fully specified machine, while a scientific null hypothesis is a choice, and a different choice produces a different probability from the same data.

Extreme means far from what the null predicts, and it has a direction, which is part of the test rather than an afterthought. If you ask only whether the filter beats a coin, you count the upper tail, and nine or more correct is the relevant region [S:69:1055]. If you ask whether it does worse than a coin, you count the lower tail, and results at least as small as yours enter the sum. If you ask whether it differs from a coin in either direction, you count both tails, and the same observation receives a larger p-value [S:69:1055]. You have to fix the direction before you see the data, or the choice becomes one more way to search.

A p-value also says nothing about the size of the improvement, and this is where headlines go wrong. A filter that catches one extra email in a thousand can produce a small p-value when the sample is large, while a large improvement can produce a modest p-value when the sample is small. The number answers a question about surprise, and usefulness is a different question.

The method has two limits that bite in practice. First, the p-value is a statement about one test, and searching produces small values by itself. With five datasets and four classifier variants each compared against a baseline, meaning a simple comparison model, you make $5\times 4=20$ comparisons. Even if none beats the baseline, you expect $20\times 0.05=1$ result that passes the threshold at the 0.05 level by chance [S:96:105]. If instead you compare all pairs among the four classifiers, the count is thirty.

The standard repair is to divide the threshold by the number of tests, or to control the fraction of false discoveries, meaning the share of significant results that are wrong, instead [S:96:105]. Second, the whole calculation depends on the null being the right kind of model, and a null that ignores dependence between emails gives a probability for a world the filter will never meet.

The formal statement. A test is a rule that maps the data to a decision. Fix a statistic and a rejection region, and reject the null when the statistic falls inside that region. The test has level $\alpha$ when, for every distribution that satisfies the null, the probability of rejection is at most $\alpha$ [S:108:233]. The p-value is the smallest level at which the data reject the null [S:108:233]. Equivalently, it is the largest probability, taken over the distributions allowed by the null, that a fresh statistic is at least as extreme as the observed one [S:116:179].

The derivation. Suppose the null names one distribution, as the fair-coin null does. Every pattern of twelve guesses then has the same probability, and the p-value is the sum of the probabilities of the patterns at least as extreme as the observed one [S:96:104]. That sum is the tail L1 adds, and the arithmetic is exact, so no approximation enters. When the null names a family of distributions, the definition takes the largest tail over the family, and the resulting number is conservative for every member. The step I omit is the argument that a largest tail exists, which needs a condition on the family rather than on the data.

The edge case. The direction of "extreme" is a choice, and it changes the number. A one-sided test counts one tail, and a two-sided test counts both. For a continuous null distribution symmetric about zero, the two-sided p-value is twice the one-sided tail. For a discrete null such as the binomial, sum the probabilities of the outcomes at least as unlikely as the observed one; doubling the one-sided tail can exceed 1 when the observed value sits at the centre, so cap the doubled value at 1 or use the exact sum [S:96:104] [S:116:179]. L1 says you fix the direction before seeing the data. The formal reason is that a p-value is defined relative to a rejection region, so trying regions until one gives a small number makes the reported number the smallest of several valid p-values. It is then no longer a p-value at level $\alpha$.

What L1 hid. L1 presented $\alpha$ as a threshold you choose. Formally $\alpha$ is a property of the test: the rejection probability stays at or below it under every distribution the null allows [S:108:233]. Choosing the threshold after seeing the data turns $\alpha$ into a description of the data instead of a bound fixed in advance. L1 was safe because it fixed the level first and used a null that named one distribution.

Where the analogy breaks. The coin analogy suggests a fully specified chance mechanism, so the p-value is one computation. A scientific null is usually composite: it says that a difference is zero, or that a coefficient vanishes, without fixing the noise. For a composite null the p-value is the largest tail over the null family, so it is a single number and does not depend on which member you happen to evaluate; it is conservative for every member [S:108:233]. If you report one member's tail instead of the supremum, then different members give different numbers from the same data.

Original sources. Dive into Deep Learning attributes the p-value and the chi-squared test to Karl Pearson, the t-distribution to Gosset, and the null hypothesis and the significance test to Ronald Fisher [S:69:1052]. It states the significance level as the risk of rejecting a true null [S:69:1053] and defines the two error types that L1 uses [S:69:1054]. The corpus presents this as a textbook summary and does not contain the original papers, so the text treats the passage as a pointer and does not quote Fisher.

An open question. The field has not settled how to do inference after selection. Barocas and colleagues note that researchers rarely run the tests needed to support a claim that one model beats another, and that a 2021 study finds many claims of state-of-the-art performance do not survive such tests [S:109:259]. The same passage says the understanding of how to account for the many sources of variance in a performance measurement is still evolving [S:109:259]. Evidence that would settle a given comparison: a test set held back until the end, a statistic fixed before scoring, and a replication on fresh rows.

A frontier result. False-discovery-rate control is the current answer when you test many things at once. The Benjamini–Hochberg procedure sorts the p-values and sets the threshold at the largest index whose p-value is at most that index times the level divided by the number of tests [S:96:105]. This bounds the expected fraction of false discoveries, and it is more permissive than dividing the threshold by the number of tests. The procedure depends on this section's idea, because every input to it is a p-value computed under its own null.

What to read next. Read Eisenstein §4.4.3 for the binomial and randomised tests [S:96:102] [S:96:104], then §4.4.4 for Bonferroni and Benjamini–Hochberg [S:96:105]. For the definition and the decision rule in a statistics text, read the chapter L1 cites [S:116:178] [S:116:179].