Logging and provenance

The authors of GPT-3 tried to remove every overlap between their training data and the test sets of the benchmarks they report. A bug meant that the removal was only partial, and retraining the model was too expensive to consider. They measured the damage instead: on one benchmark, 29 per cent of the examples were flagged, and the cleaned subset scored 3 percentage points lower [S:232:31-32]. The result stayed in the paper with an asterisk beside it, and the bug stayed in the paper as a sentence [S:232:31].

This section asks how you know what you ran. You know it when the record is complete enough that a stranger can rebuild the run without asking you a question, and when the record was written by the program rather than by your memory. The GPT-3 episode is the argument in one line: a fact discovered after the run could be reported only because somebody could still measure it. The disclosure was possible because the authors still held the analysis, and not because anybody remembered the details of the run. !A record of one run, built from six fields: the seed, the data with its size and fingerprint, the split sizes and the seed that drew them, every setting including the ones held fixed, the code version, and the number of configurations tried. In the worked example the seed and the batch size are missing, each open over ten values, so a hundred different runs would produce the same record. Each open field multiplies the count, and the record is decisive only when the product is one. Status: schematic.

The idea. Provenance is the record of where data, code and results came from and how they were produced. Reproducibility is the property that another person can rerun the procedure and obtain the same result (simplified; the precise statement is in c04.s04 D1). The record is the object that makes the second possible, and it is written once, at run time.

Six fields carry most of the weight. Record the seed s, the data with its size and a fingerprint, and the split sizes together with the seed that produced them. Then record every setting, including the ones you did not vary, the code version, and the count of configurations you tried before this one. Two more decide whether the six mean anything: the metric with the subset it is computed on, and the software stack — library versions, hardware and thread count.

The count is an input to the decision, not the decision: Bonferroni's principle says to calculate the expected number of bogus results and to distrust the finding when that number swamps the real ones you hope to find [S:57:26]. The record should carry the count so that the calculation can be made. Published checklists for machine-learning papers now ask authors to report several of these fields — code and data, data splits and hyperparameters, error bars over random seeds, and the compute used — although the practice is not universal [S:254:15]. The checklist does not ask for the split seed or the count of configurations tried.

The split deserves its own line because a conclusion is only as good as the independence of the two piles. Drawing sensible conclusions from a learning experiment requires the result to be independent of which items landed in the training set and which in the test set [S:109:258]. If the record does not say which items those were, nobody can check that independence, including you.

One caution belongs in the record itself. The same seed does not guarantee the same number: floating-point addition is not associative, so a change in the order of operations can change the last digits [S:2:27]. A record that promises identical digits across machines promises more than the arithmetic can deliver.

Worked example. Suppose the record lists the dataset, the split of 600, 150 and 150 items, the code version, the metric with its subset, and the score of 0.91 on that metric. It does not list the seed, and it does not list the batch size, and the sweep used ten values of each. How many settings must a stranger try before finding the run that produced 0.91?

$$\text{settings to search}=10\times 10=100$$

The sentence behind the equation matters more than the equation itself. Each field left open multiplies the settings a stranger must try, and the score tells them when to stop. Two open fields over ten values each give a hundred settings to search. If no two of them round to 0.91, the score identifies the run, and a record that prints two decimals is decisive only when the scores do not collide.

Write it while the run happens. A number copied from a terminal into a document is a claim about the past; a row written by the harness with its inputs is a record. The difference shows up six months later, when the arbitrary choices are gone from memory and the only surviving evidence is what the program wrote down. A record is not a diary, either: it contains the values that the result depends on, and a description of how the work felt is not provenance. The harness described in the next section is the natural place to write the values, because the harness is the only part of the process that sees all of them.

A run that was never recorded cannot be reconstructed at all, since the arbitrary choices that produced it have left no trace anywhere.

The limitation is that a complete record makes a run repeatable rather than correct. A leak, a mislabelled column or a wrong metric reproduces perfectly, and the record will faithfully describe an experiment that measured nothing. Provenance tells you what you did; the earlier sections of this chapter tell you whether it meant anything.

The formal statement. A run is a point in a product space with one coordinate per setting. Each coordinate admits a finite number of values. A record fixes some coordinates and leaves the rest open. The number of settings a stranger must search is

$$\text{settings to search}=\prod _{\text{open fields}}\left(\text{values the field admits}\right)$$

Read it as: multiply the number of values that each unrecorded field could take. The record also carries the score, which filters the search: a candidate run survives only if it reproduces that score to the printed precision. The product is therefore an upper bound on the settings to try, not a count of runs that are consistent with the record. L1's 10 × 10 = 100 is this bound with two open fields.

The derivation. An open field contributes its count independently of the others, and independent counts multiply. The hypothesis is independence, and in practice it fails. A step count and a batch size have to agree with the dataset size; a learning rate and a schedule are chosen together. Where fields are linked, the product overcounts, so it is an upper bound on the settings to search rather than an exact count. The score narrows it further: when no two candidates round to the printed value, the score identifies the run; when several do, the record still hides which one ran. A product of ten and ten is far from one, which is why the record is not decisive on its own.

Reproducibility, precisely. L1 says another person obtains "the same result". Arithmetic forbids the strong reading. Floating-point addition is not associative, so the last digits depend on the order of the operations, and that order changes with the code path [S:2:27]. Reproducibility is therefore agreement to a stated tolerance on named quantities, and the tolerance belongs in the record beside the score. With a tolerance the relation is not transitive. Three numbers can each sit within tolerance of the next while the first sits outside tolerance of the third. A report that says "reproduced" and names no tolerance has said less than it appears to.

The edge case. A record can be complete over every field you listed and still fail to identify the run. The coordinate that varied was then one you did not think of: the thread count, the library version, the order in which files were read. Completeness is relative to a schema, and the schema must be written before the run. L1's six fields are a starting list and not a definition.

What L1 hid, and why it was safe. L1's worked example treats the seed as the decisive field. For the ten-run table of 4.1 it is decisive, because nothing else changed. Across two machines it is not decisive, and the record has to say so.

Why the record is written first. Memory is not a record. A person can rebuild the gist of a procedure a week later, because memory keeps the deviations that mattered; a run leaves no such trace. The process is gone and the only witness is the file. "Roughly right" is meaningless for a witness that no longer exists, which is why the record is written while the run happens rather than afterwards.

Original sources. The modern statement of the problem is GPT-3's contamination section. The authors tried to remove all overlap between pretraining data and benchmark test sets, a bug made the removal partial, retraining was too expensive, and they measured the effect instead [S:232:31]. The disclosure is possible only because the analysis survived the run. The practice of documenting a dataset's creation, composition and preprocessing is the datasheet proposal [S:109:262].

An open question. What a record must contain to make a contamination claim checkable is unsettled. GPT-3's authors call contamination detection a new area without established best practices [S:232:29]. They also name the limit of their own analysis: they cannot be sure the clean subset is drawn from the same distribution as the original [S:232:33]. Llama 3 states the same problem for detection methods, which disagree with one another [S:431:33]. Evidence that would settle it: a record schema fixed in advance, with the contamination rule and its threshold stored beside the score, so that two labs can compare numbers without re-deriving the rule.

A frontier method. Llama 3's audit is the current shape of such a record. An 8-gram overlap rule, a threshold chosen per dataset, and an estimated performance gain printed beside each benchmark [S:431:34]. It is provenance at the level of a corpus rather than a run, and it is what makes a claim about a benchmark score checkable by a reader who does not hold the model. A reader can then ask whether the reported score survived the rule, which is the minimum a provenance claim has to support.

What to read next. Read the contamination section of [S:232:31] through [S:232:33], because the sequence from bug to measurement to caveat is the argument of 4.4 in one document. Then read [S:109:262] for datasheets, which is the schema half, and [S:431:34] for the audit half.