A result and its luck

Ten runs of one program, on one dataset, gave the ten numbers in this table. The program assigns 150 held-out items to one of two classes, and between runs one thing changed: the seed.

!Ten runs of one program on the same 150 held-out items, differing only in the seed. The runs score 8, 9, 9, 10, 10, 11, 12, 12, 13 and 15 errors, so the shaded band is seven errors wide, about 4.7 per cent of the test set. The dashed line at 10.9 is the mean, and no run produced it. The points are not joined, because the run number is a label and not a quantity. Status: synthetic.

The best run produces 8 errors and the worst produces 15, from the same code and the same data. The number you would record in your notebook depends on which of the ten runs you happened to perform, and the notebook would not record that choice. This section asks how much one run tells you. A single run tells you that one result is possible, and it says nothing about the range of results that the same procedure produces.

The chapter asks how far a result can be trusted, and this table supplies the first part of the answer, which is that one number is the least informative summary of a procedure that has a spread.

The idea. An experiment is a procedure that produces a result and can be repeated under recorded conditions. What makes a run repeatable is the random seed, the number that fixes every arbitrary choice the program makes; write it s. The generator computes a mathematical function, so the same input twice produces the same output, and that is why the seed goes into the record [S:13:334]. Resetting the generator to a recorded value reproduces the sequence exactly, which is the property that testing and debugging depend on [S:13:339].

Repeatability and typicality are different properties, and the seed gives you only the first of them. The seed is not a feature of the problem, and a better understanding of the problem does not make one seed more representative than another [S:38:130]. The useful strategy is therefore to vary the seed on purpose, run the procedure B times, once for each r from 1 to B, and retain every result. The procedure is reproducible and not representative.

The tempting repair is to report the mean of the ten runs, which takes one line of arithmetic: 109/10 = 10.9 errors. That is honest arithmetic and an unsatisfactory summary, because 10.9 is a number that no individual run produced. Averaging treats the seed as a random variable that has to be marginalised, and the seed is not that kind of object at all [S:38:130].

A better report gives the whole collection, the smallest and largest values, the seeds that produced them, and B. The observed band is a lower bound on the spread the procedure can produce, and it rises with B, so report B beside it. Use at least five runs and record why you chose that number. SAC trains five instances of each algorithm with different random seeds and shades the band between the minimum and the maximum return over those trials [S:291:7]. The picture that results is a range rather than a point, and a range is what the procedure actually has.

A method is not its best run, and it is not its mean run either. The list of B numbers is the object you possess, and any single number quoted from that list is a choice the author made and the reader never saw.

A single run is like one photograph of a landscape taken through a window that opens for a moment. It shows you something real and tells you nothing about the range of weather that the place produces. The comparison breaks because a photograph cannot be wrong about what it shows, while a run can come from a bug, and then the number is not a sample of anything.

Worked example. Three calculations to write beside the table. The sum is 8+9+9+10+10+11+12+12+13+15 = 109, so the mean is 109/10 = 10.9 errors. The smallest run produces 8 errors and the largest produces 15. Their gap, as a fraction of the 150 test items, is:

$$\frac{15-8}{150}=\frac{7}{150}\approx 0.047$$

Seven errors in 150 separate the best run from the worst, which is about 4.7 percentage points. A paper that printed one of these runs and called it the error rate would be reporting a point inside a band of that width, without telling the reader that the band exists.

Reporting every run demonstrates the spread and gives no criterion for deciding whether a difference between two procedures is real. Researchers rarely perform the statistical tests needed to support a claim that one model beats another, and the understanding of how to account for the many sources of variance in these measurements is still developing [S:109:259]. That judgement requires probability, which Chapter 15 supplies, and this section stops short of it on purpose.

One more limit belongs here, and it is easy to miss because it hides inside the word "same". The seed pins the sequence that the program draws, not the arithmetic that the machine performs with those numbers. Floating-point addition is not associative, so a change in the order of operations can change the last digits and the reported number [S:2:27].

The formal statement. Fix the data, the code and every setting. The harness then defines a function from seed to result: each seed $s$ returns one value of the quantity the run measures. Write ${\hat{R}}_{\text{train}}(s)$ for the average loss on the training items, ${\hat{R}}_{\text{test}}(s)$ for the error on the held-out items — the quantity in the 4.1 table — and $S(s)$ for the higher-is-better score used in 4.3. A run is one value of that function. The report is the indexed family of values over $B$ enumerated seeds, written $s_1,\dots ,s_B$, one per run index $r$ from 1 to $B$. Three hypotheses make the description exact: the harness is deterministic given its inputs, nothing outside the seed changes between runs, and you chose the seeds.

The third hypothesis carries the weight. An enumeration is not a sample, so no distribution over seeds comes with the problem. That is what L1 means when it says the seed is not a feature of the problem [S:38:130].

The spread. Define it as:

$$\text{spread}=\underset{1\le r\le B}{\max }{\hat{R}}_{\text{test}}(s_r)-\underset{1\le r\le B}{\min }{\hat{R}}_{\text{test}}(s_r)$$

Read it as: the gap between the worst and the best held-out error over the seeds you ran. This number needs no distribution over seeds, and that is its advantage.

A mean is a different object. The average of the $B$ values is defined by arithmetic and needs no distribution. What needs a distribution is the claim that it estimates the procedure's expected result: that reading treats the seeds as draws from a population of seeds, and the enumeration supplies no such population, because the seeds are a design choice [S:38:130]. The uniform weights are a choice, not a property of the procedure. L1 said that averaging marginalises an object that should not be marginalised [S:38:130].

The proof, and the step omitted. The range needs no proof. The standard error of the mean needs independent draws, and a variance estimated from independently sampled datasets is exactly what the classical analysis assumes [S:74:146]. Your enumeration does not supply that independence. Two runs are correlated when they share random inputs that the seed does not vary — the same data split, the same initialisation, the same environment stream. Shared code and hardware are a common deterministic component and shift every run equally; they do not correlate the deviations. The across-seed standard error therefore omits the variance components the seed does not control [S:38:129], one of the "numerous sources of variance" the fairness book says are still poorly accounted for [S:109:259]. The correction needs a model of which choices the seed actually controls, and no such model is available from the run alone.

The edge case. Suppose one row in the list came from a mislabelled column. The list is then not a family of results from one procedure. It mixes the procedure with a defect, and the spread measures the defect as much as the seed. The band is honest about the numbers and misleading about the method. L1's photograph comparison already allows for this, and the repair is the sanity checks of 4.5 rather than a wider band.

What L1 hid, and why it was safe. L1 wrote "the same code and the same data". Reproducibility is conditional on the machine as well. Floating-point addition is not associative, so a different reduction order gives different last digits [S:2:27]. That condition was safe in L1 because the ten runs came from one machine.

Where the analogy breaks. A photograph records a moment that cannot be re-entered, and the weather between shots is outside the photographer. In the harness nothing outside the seed changes. The variation is inside the procedure, produced by it, and reproducible on demand. The photograph samples the world; the run is a coordinate of the code.

Original sources. The two halves of the idea sit in different literatures. Structure and Interpretation of Computer Programs supplies the property that makes a seed useful: the generator computes a function, so the same input twice produces the same output [S:13:334]. Resetting the generator to a recorded value reproduces the sequence [S:13:339]. Probabilistic Numerics supplies the argument about typicality: averaging over seeds treats the seed as a random variable that must be marginalised, while a seed is a design choice and not a feature of the problem [S:38:130]. The empirical case arrived with deep reinforcement learning, where several studies found the choice of seed significant for popular algorithms [S:38:129].

An open question. Whether to average over seeds or to treat each seed as a coordinate is unsettled. The same pages argue against averaging, on the ground that a pseudo-random generator makes performance an unpredictable function of the seed, so the seed direction can never be improved [S:38:130]. Practice runs the other way, and the fairness literature notes that the statistical tests needed to support a claim that one model beats another are rarely performed [S:109:259]. Evidence that would settle it: a variance decomposition on a fixed benchmark, reporting the share of the spread between two methods that comes from seeds, from data splits, and from their interaction. The decomposition has been published: the 2021 paper the fairness book cites at this point is Bouthillier et al., "Accounting for variance in machine learning benchmarks" [S:109:292], and [S:109:259] reports its finding that many state-of-the-art claims do not survive such tests. What is open is how to standardise the decomposition across benchmarks, not whether one exists.

A frontier result. The display convention has moved further than the inference. SAC trains five instances of each algorithm with different seeds and shades the region between the minimum and the maximum return over those trials [S:291:7]. That is a range, honestly drawn, and it is not an interval with a coverage guarantee.

What to read next. Read [S:38:129] and [S:38:130] as one argument, because the empirical claim and the case against averaging pull in opposite directions. Then read [S:109:259] for what the field does instead of testing, and [S:13:334] for the property the whole practice rests on.