Leakage and contamination

A much-repeated story from computer vision concerns a classifier, meaning a model that sorts rows into categories, that was said to tell Russian tanks from American ones with high accuracy. The trick was the weather: the Russian tanks had been photographed on cloudy days and the American tanks on sunny days [S:109:259]. The source calls the story apocryphal, meaning nobody can trace it to a real study, which is the right label, and the lesson survives the label.

The model had learned the sky rather than the tanks, and the sky would not appear in the same way when the model was deployed, that is, put to work on new rows. Leakage is the name for a spurious relationship between the inputs and the answer that comes from how the data were collected or sampled [S:109:259]. It inflates the score, because the relationship is real in the table and absent in the world the model will meet [S:109:259]. The section question is how a test set becomes a training set by accident, and the short answer is that information about the answer can reach the model through the inputs rather than through any genuine predictive pattern. !Three routes by which a test set becomes a training set by accident. In (a) the column is recorded after the outcome: the discharge date follows the readmission decision, and in the section's six-patient example a test-ordered column of the same shape scores 6/6 where the honest rule scores 4/6. In (b) the rows overlap: benchmark text sits inside a web crawl, and decontamination removes the match without proving the overlap gone. In (c) the rows are ordered and split at random, so the model trains on later rows and is scored on earlier ones, a score deployment cannot reproduce. The figure is schematic. Status: schematic.

One route runs through a single column. A hospital records whether each patient was readmitted, and it also records the date of discharge, which is set after the readmission decision and sometimes because of it. A model that uses the discharge date can score well while learning nothing about medicine. The column is not wrong, and its values are not corrupted; it is recorded at a time that makes it a consequence of the answer.

The other route runs through the rows. Contamination is the presence of evaluation rows, or text very close to them, in the training data [S:85:192]. It is a form of leakage, and a common cause is a sweep of the public web that copied a benchmark, meaning a fixed set of rows and a scoring rule that many teams use, along with everything else [S:85:192]. Decontamination searches the two sets for matching text and removes the overlaps, which reduces the damage without proving that the damage is gone [S:85:192].

Time is the cleanest case of the same mistake. When rows arrive in order and you split them at random, the model trains on the future and is tested on the past, a phenomenon called look-ahead [S:21:276]. The score looks good, and deployment will not reproduce it, because deployment always moves forward from the data you have [S:21:276].

Benchmarks are not immune, and the administrators know it. Contest organisers restrict how often the test set behind a public ranking, or leaderboard, may be scored, because repeated submission turns that test set into a tuning set [S:109:259]. The restriction admits that a test set stops being a test set once decisions are made with it. The same logic applies inside one project, where a score quoted in a meeting becomes a target for the next week of work.

Contamination has a second source that is easy to miss, namely the model itself. Benchmarks are often listed on public web domains that are crawled, and users pass questions into models that later become training data for future models [S:85:192]. The rows were never meant to be training rows, and no one chose them, which is why the problem recurs across generations of models.

Worked example. Six patients, one column that records whether a lab test was ordered, and one label that records whether the patient was ill.

A rule that copies the test column gets every patient right. Drop the column and the best remaining rule answers yes for everyone, because four of the six patients were ill, and it gets four right. The two scores are far apart.

$$\text{with the column}=\frac{6}{6}=1.0,\text{without it}=\frac{4}{6}\approx \mathrm{0.67.}$$

The gap of 2/6 is not a small correction; it is the difference between a perfect model and a mediocre one. A statistical test compares a number computed from the data with what chance alone would produce. On this table alone no such test separates the column from a genuine predictor, because the table is consistent with both stories: the column may record a consequence of the answer, or the answer may follow from the column. A test can compare the table with a fresh sample from deployment and detect that something has shifted, but it cannot say which column caused the shift. You find the leak by asking, for every column, when the value was recorded and who already knew the answer at that moment [S:109:259].

Leakage resembles an exam where the supervisor leaves the answer sheet on the desk. The comparison breaks because an answer sheet is obvious, while a leaking column looks like an ordinary feature and its numbers can be perfectly innocent. That is why leakage survives code review and disappears only under a question about provenance.

The idea has a hard limit, and the limit is practical rather than conceptual. Leakage is invisible in the final number, so a clean score is not evidence that the data are clean. Removing a suspicious column can also remove genuine signal, and some columns that look like leaks are part of the task definition, such as a diagnosis that a clinician would legitimately have at prediction time. Decontamination is a search over text, not a proof of independence [S:85:192].

The formal statement. Let the table be a sample from a joint distribution over inputs and labels, and let the population the model will meet have its own joint distribution. A column leaks when its relationship with the label is an artefact of how the rows were collected or sampled, so the relationship will not hold when the model is deployed [S:109:259]. That is narrower than a difference between the input distributions: a column can leak while the two distributions look alike, if the column is recorded after the answer is known, and a distribution difference can carry label information without being an artefact. Barocas, Hardt and Narayanan define leakage as a spurious relationship between the feature vector and the target that is an artefact of collection or sampling [S:109:259]. There are three channels. A column can be recorded after the answer is known. A row can be copied from the evaluation set into training, which is contamination [S:85:192]. A model can be selected by repeated queries to the same test set, which is adaptive overfitting [S:109:256].

The derivation. Take the column route. If the column is a deterministic function of the label, a rule that copies the column is correct on every row, so its score is perfect and says nothing about the world. That is the arithmetic of L1's six-patient table: perfect with the test column, and no better than the majority rate without it. Nothing about medicine enters, so nothing about medicine can be inferred from the score.

The edge case. L1 tells you to ask, for every column, when the value was recorded. That audit finds column leaks and misses row leaks entirely, because a contaminated row arrives with innocent columns and a plausible label. Operationally, decontamination searches the two sets for n-gram overlap or for fixed-length character matches [S:85:192]. The search is evidence of overlap and not proof of independence, because the same problem can reappear with different numbers, a case the RLHF book records [S:85:192].

What L1 hid. L1 said that no test on the table alone finds the gap. The formal reason is that a leak is a statement about how this table was produced, and a test computed inside the table has no access to the production process. A test against deployment rows can detect that something has shifted, but not which column caused the shift, so provenance remains the direct check. L1 was safe because every example was a column leak inside a single table.

Where the analogy breaks. The answer-sheet analogy suggests that a leak is an extra object you can see and remove. Formally, a leak is a property of the joint distribution, so dropping the column can remove genuine signal and leave the distribution still shifted. The analogy also suggests a clean inspection, while a leaking column has ordinary values and a plausible name, and only its provenance distinguishes it [S:109:259].

Original sources. Barocas, Hardt and Narayanan give the definition L1 uses and point to Kaufman and colleagues for an overview of leakage in machine learning [S:109:259]. The corpus does not contain that overview, so the text cites the pointer and does not attribute claims to it. The contamination side has a recent textbook statement: the RLHF book defines dataset contamination as a form of leakage and describes decontamination as a search for n-gram overlap over words or subword tokens, or for fixed-length character matches [S:85:192].

An open question. The field has not settled how much repeated use of one test set corrupts a score. Barocas and colleagues state the worry that incremental use of a test set produces adaptive overfitting [S:109:256]. They then report that the effect has not appeared at the scale predicted: models that do better on the old test set also do better on a fresh one [S:109:257]. They add that the theoretical understanding of why practice has not overfitted is still catching up [S:109:257]. Evidence that would settle it: a fresh test set collected after the fact, scored once, with old and new rankings compared, as the replication reported there did [S:109:257].

A frontier result. Contamination-limited benchmarks answer the problem by construction. LiveBench draws questions from recent sources, updates them monthly and scores answers against objective ground truth, so memorisation is unlikely for questions drawn from sources newer than a model's training data [S:636:1]. The design depends on this section's idea, because it treats the collection date of a benchmark as part of what the benchmark measures.

What to read next. Read the leakage passage in Barocas and colleagues for the definition and the tank story [S:109:259], then their benchmark chapter for adaptive overfitting and the replication result [S:109:257]. For contamination, read the evaluation chapter of the RLHF book [S:85:191] [S:85:192]. Then read the LiveBench paper for a benchmark built to resist it [S:636:1].