Sections 4.1 and 4.4 leave a question open. Section 4.1 says a single score is one point from a spread, and that the understanding of how to account for the many sources of variance in these measurements is still developing. Section 4.4 says a record with an open field is consistent with many runs, and that a complete record makes a run repeatable rather than correct. The chapter does not settle what a report of a machine-learning experiment must contain for a stranger to decide whether a difference between two procedures is real.
Propose a minimum reporting standard for one experiment that compares two procedures. In your answer: name the fields the record must contain and say what each field protects against; say how the report should display run-to-run variation; say what the standard should require when the difference between the procedures is smaller than that variation; and name the evidence that would show your standard is too weak or too strong.
Rubric. Tick each criterion your answer meets, then save the self-assessment. The platform does not grade it.
Worked solution
There is no single correct answer, so this is a model of the shape a complete answer takes, not a key.
A complete answer starts with the record. The six fields from section 4.4 are the seed, the data with its size and fingerprint, the split sizes and the split seed, every setting including the ones not varied, the code version, and the count of configurations tried. Each field removes a way for the same reported number to stand for different runs. The seed removes the arbitrary choices inside the program. The data fingerprint removes a silent change of the input. The split removes the question of which items were held out. The settings remove the defaults that the author did not write down. The code version removes the difference between two programs with the same name. The count of configurations removes the search that produced the winner. A field that does not change the number of runs consistent with the record is optional; a field that does belongs in the standard.
The display rule follows from section 4.1. The report shows every run, or the smallest and largest with the seeds that produced them, and it places that band beside the difference between the two procedures. A mean alone is a summary of the runs and not a run; a standard deviation alone hides the shape and the extremes. The band is the cheapest display that keeps the spread visible, and a plot of every run is better when the number of runs is small.
The rule for a difference smaller than the variation is the hard part. A defensible standard says: do not call the difference a finding; report the band, report the difference, and say that the two overlap. It may allow the authors to report the difference as a candidate for a larger experiment, with the seeds and the count of configurations that produced it. It should not allow a bar chart of two means with no spread, because that display makes a small difference look like a large one. The standard can require the sentence; it cannot require the authors to believe it.
The standard has limits. It can make a run repeatable and it cannot make it correct: a leak, a mislabelled column or a wrong metric reproduces perfectly, as section 4.4 says. It also cannot settle the many sources of variance that section 4.1 says the field is still learning to account for. A standard that required a full statistical test would be stronger and would fail more often, because the test needs assumptions the record does not supply; a standard that required only the seed would be cheaper and would leave the search and the split invisible. The evidence that would make this standard too weak is a published comparison that satisfies every field and still cannot be reproduced; the evidence that would make it too strong is a field that costs more to record than the decision it protects. The count of configurations is the field most likely to be omitted and the one that most changes the reading of the best result.