A one-file experiment harness

A Street View transcription system was reading address numbers badly, and nobody could see why from the score alone. Sorting the images by how confidently the system got them wrong exposed a crop that cut digits off the edge, and fixing the crop improved the entire system [S:66:397]. The fault appeared as a pattern in the most confident mistakes, and not as a single bad number, which is the lesson this section turns into a program.

This section asks what a reproducible experiment looks like: one file that takes a written configuration, performs every step in a fixed order, and prints one row per run. Nothing important lives in a notebook cell, in a chat message or in somebody's memory, because none of those can be re-run. !A dataset of 900 items divided into 600 for training, 150 for validation and 150 for testing. The bar is cut twice and the parts are disjoint, so every item sits in exactly one part: 600/900 + 150/900 + 150/900 = 2/3 + 1/6 + 1/6 = 1. The fractions are the arithmetic form of that sentence, and a split whose fractions do not add to one has lost or duplicated an item. Status: schematic.

The idea. A harness is one program that runs the whole experiment, from loading the data to writing the result, and it is the only part of the work that sees every value. Reproducibility is the property that another person can run the harness and obtain the same row. The harness has a contract in the sense of Chapter 1: what it accepts, what it returns, what it leaves unchanged, and what it does not promise. !Ten runs of the harness, one row each. Every row carries the score and the settings that produced it, and the seed is what ties the row to its run. Delete the seed column and the ten rows remain, but nothing in the file says which run produced which row, so the file can no longer be rebuilt. The row count checks that no run was silently overwritten; the seed column checks that each row still points at a run. Status: schematic.

The four lines are worth writing out. It accepts a configuration file and a data file, it returns one row per run containing the score and the settings that produced it, it leaves the data file untouched, and it does not promise that the score is meaningful. The last line is the one that keeps the harness honest, because a program that runs a leaking experiment runs it perfectly.

What the harness must not do is decide. It should not choose the seed, the split or the metric silently, because a program that fills in a default has made a decision that its author will not find in the record later. Every value is either in the configuration file or printed in the row.

The split is a partition. Take a dataset of n = 900 items divided into 600 for training, 150 for validation and 150 for testing. The three parts are disjoint and their union is the whole dataset, which is a claim you can check rather than assume.

$$\frac{600}{900}+\frac{150}{900}+\frac{150}{900}=\frac{2}{3}+\frac{1}{6}+\frac{1}{6}=1$$

The fractions add to one, and that is the arithmetic form of the sentence "every item is in exactly one part"; a proof needs two lines. The invariant says that each item processed so far sits in exactly one part, and the measure says that the number of unplaced items falls by one per step. The loop therefore ends with no item unplaced, which is exactly the property that the split claims. Chapter 3 supplies both tools, and this is the one theorem in this chapter that you can prove outright.

Three tests the harness runs. Fit a tiny dataset first, because a pipeline that cannot label a single example has a software defect and not a hard problem [S:66:397]. Plant the answer second, by adding a feature worth +100 for positive items and −100 for negative ones, and check that the model can now fit the training data; if it cannot, you have a defect rather than a difficult task [S:47:71]. Shuffle and resplit third, mixing the training and test items and drawing a new test set, because a large change in the score means the two piles were never from the same pool [S:47:70].

Worked example. Run the harness ten times with ten seeds. Each run writes one row, so the output file has ten rows, and the tenth row can be checked against the run that produced it. Now delete the seed column and try to rebuild the file: the rows remain, the runs behind them do not.

The arithmetic of the split and the arithmetic of the rows answer different questions. The fractions check only that the three sizes add up: if you lose one item and duplicate another, the sum is still one. The check that catches loss and duplication is a set check: build the three index sets, verify that no item id appears in two of them, and verify that their union has 900 members. The row count checks that no run was silently overwritten. All three are inexpensive to compute, and each catches a failure that a single printed score hides.

A recipe card that records the oven setting as well as the ingredients is the usual picture of a harness. A recipe still works in a different oven; the harness's promise of an identical row holds only for the same code, the same data and arithmetic that rounds the same way.

The limitation is that a harness makes an experiment repeatable rather than honest. It will reproduce a leak with perfect fidelity, and it will reproduce a wrong metric with the same fidelity. Fixing the order of the steps is necessary and it is not sufficient, which is why the earlier sections come before this one.

The formal statement. Let $D$ be a finite family of $n$ items. A split is a triple of subsets, for training, validation and test, that are pairwise disjoint and whose union is $D$. The harness contract has four clauses: it accepts a configuration file and a data file, it returns one row per run, it leaves the data file unchanged, and it makes no claim that the score means anything. The first three are properties a test can check.

The proof. Place items one at a time. The invariant is that every placed item sits in exactly one part. The measure is the number of unplaced items, which falls by one per step, so the loop terminates with none left. Hence each item sits in exactly one part, and the parts are pairwise disjoint. For the union, count: the sizes of the three parts sum to $n$, the parts sit inside $D$, and $D$ has $n$ items, so the union exhausts $D$. Chapter 3 supplies the two tools, and the only hypothesis is that each step places one item exactly once [S:47:65].

The edge case. Drop that hypothesis and the theorem fails. Assign each item to each part by an independent draw with probabilities $2/3$, $1/6$, $1/6$ — three draws per item. The expected membership of each item sums to one, so the expected fractions still sum to one, but an item can land in two parts or in none, and the realised triple is not a partition. A single draw of one part per item is a partition by construction; so is a fold index computed from the item's position ($n\mathrm{mod}K$) [S:47:65]. ESL states the same split as K roughly equal-sized parts [S:64:260]. Shuffling and dealing is a partition too; independent assignment per part is a distribution over subsets.

The second edge case. A harness that prints one row per run cannot distinguish "did not run" from "ran and crashed". The row count is then not a count of runs, and the missing row is the one you need most. A status field written at the start and updated at the end turns an absence into a record.

What L1 hid, and why it was safe. L1's fractions use 600, 150 and 150 of 900, and those counts divide exactly. With 901 items the same three-way split cannot keep the shares exact, and the sum still equals one only because the record states counts rather than rounded shares. L1 was safe because its numbers were chosen to divide.

Where the analogy breaks. A recipe card travels. The same card works in a different oven, because it describes a process that tolerates small changes. The harness promises a row, and the row holds only for the same code, the same data and arithmetic that rounds the same way [S:2:27]. A recipe promises a dish and the harness promises digits, so the card's portability is not the property being claimed.

Original sources. The one-file harness is a convention rather than an invention, and the corpus gives its pieces. Deep Learning states the tests that separate a defect from a hard problem: fit a tiny dataset, and check that a model can overfit a planted feature [S:66:397]. A Course in Machine Learning gives the shuffle-and-resplit test for a train/test mismatch [S:47:70] and the cheat feature that a working learner must be able to use [S:47:71]. The same chapter frames debugging as a search for where error enters the pipeline, which is the harness's reason to exist [S:47:70].

An open question. Whether a fixed harness plus a seed is enough for another lab to obtain the same row is unsettled at the level of arithmetic. Floating-point addition is not associative, and the order of operations changes with parallelisation, so bitwise agreement across machines is not guaranteed even for identical code [S:2:27]. The practical question is what tolerance a reproduction claim should name. Evidence that would settle it: a cross-machine study on a fixed harness, reporting the spread of differences between runs under a stated thread count, compiler and library version. Until then, "reproduced" is a claim with an unstated parameter.

A frontier method. Terminal-Bench treats the harness as an audited artefact. Each task ships a container, an oracle solution that must pass, a dummy solution that must fail, and a contributor checklist that includes confirming reproducibility [S:505:4]. That is the harness of this section turned into an entry requirement, and the dummy solution is the planted-answer test of L1 in another dress.

What to read next. Read [S:66:397] for the tiny-dataset test and [S:47:70] for the resplit test, then write both into one file before reading anything else. Then read [S:2:27] for why the row will not match to the last digit, and [S:505:4] for what a harness looks like when it must survive an audit.