Derive: Experiments That Do Not Lie

A binary column is a string of 10 bits. Fix the label string. A column that agrees on exactly $k$ positions agrees at those $k$ and disagrees at the remaining $10-k$. Choose the $k$ positions where it agrees in $\left(\genfrac{}{}{0ex}{}{10}{k}\right)$ ways; once the positions are chosen, every entry of the column is forced. The number of columns agreeing on exactly $k$ items is therefore $$\left(\genfrac{}{}{0ex}{}{10}{k}\right)=\frac{10!}{k!(10-k)!}.$$ The counts for $k=10$, $9$ and $8$ are $$\left(\genfrac{}{}{0ex}{}{10}{10}\right)=1,\left(\genfrac{}{}{0ex}{}{10}{9}\right)=10,\left(\genfrac{}{}{0ex}{}{10}{8}\right)=45.$$ The number agreeing on at least 9 items is $$\left(\genfrac{}{}{0ex}{}{10}{10}\right)+\left(\genfrac{}{}{0ex}{}{10}{9}\right)=1+10=11.$$ As a fraction of all $2^{10}=1024$ columns, $$\frac{11}{1024}\approx 0.01074,$$ which is about 1.07 per cent. The count is the reason a search over columns can look impressive on a small table: even when the labels are random, 11 columns agree on 9 or 10 of the 10 items. Eleven candidates is enough for a selection step to find one, and the winning column then carries the labels across the split. Section 4.2's six-item example has $\left(\genfrac{}{}{0ex}{}{6}{6}\right)+\left(\genfrac{}{}{0ex}{}{6}{5}\right)=1+6=7$ columns that agree on at least five of six; for the same threshold of at most one miss, the count grows with $n$, and the fraction of near-perfect columns falls, but the absolute number rises.

Take two groups of two items, four items in total. Put one item from the first group and one from the second into training, and the other two items into test. Each item is in exactly one part, so the split is a partition. The fractions are $$\frac{2}{4}=\frac{1}{2}\text{and}\frac{2}{4}=\frac{1}{2},$$ and they sum to one. Every test item shares its group with a training item: the test item from group 1 has a training partner in group 1, and the test item from group 2 has a training partner in group 2. The partition check passes and the independence condition fails. The test part contains no group that the training part has not seen, so the number of new test groups is 0.

The example uses the smallest dataset with at least two groups. A group of size one cannot put an item on both sides, so every group must have at least two items. Two groups of two items give four items, and the mixed 2/2 split above is one split of that dataset with nonempty parts that shares a group across the boundary. Section 4.2's photographs are the same structure at a larger size: a thousand photographs of ten thousand pixels are a thousand groups of ten thousand pixels, and a split by pixel passes the partition arithmetic while the pixels of one photograph travel together. The partition says that no item was lost or duplicated. It says nothing about which groups the items came from.

Let $f_1,f_2,f_3$ be the full-system scores, $a_1,a_2,a_3$ the ablated scores on the same seeds, and $a_1^{\prime },a_2^{\prime },a_3^{\prime }$ the ablated scores on the second seed set. The paired removal effect is $$D_p=\frac{1}{3}\sum _{s=1}^3(f_s-a_s)=\frac{1}{3}\sum _{s=1}^3{f}_s-\frac{1}{3}\sum _{s=1}^3{a}_s=\bar{f}-\bar{a}.$$ The unpaired effect is $$D_u=\bar{f}-{\bar{a}}^{\prime }.$$ Subtracting gives $$D_u-D_p=(\bar{f}-{\bar{a}}^{\prime })-(\bar{f}-\bar{a})=\bar{a}-{\bar{a}}^{\prime }.$$ The difference between the two effects is exactly the difference between the mean ablated scores on the two seed sets.

For the numbers, $$\bar{f}=\frac{80+84+76}{3}=\frac{240}{3}=80,$$ $$\bar{a}=\frac{60+66+58}{3}=\frac{184}{3},{\bar{a}}^{\prime }=\frac{64+70+62}{3}=\frac{196}{3}.$$ The paired effect is $$D_p=80-\frac{184}{3}=\frac{240-184}{3}=\frac{56}{3}\approx 18.67,$$ and the unpaired effect is $$D_u=80-\frac{196}{3}=\frac{240-196}{3}=\frac{44}{3}\approx \mathrm{14.67.}$$ The difference is $$D_u-D_p=\frac{44}{3}-\frac{56}{3}=-\frac{12}{3}=-4,$$ which agrees with $\bar{a}-{\bar{a}}^{\prime }=184/3-196/3=-4$. The second seed set gives the ablated system 4 points more on average, so the unpaired comparison understates the removal effect by 4 points. The component looks less valuable than it is, and the gap is a property of the seeds rather than of the component.

The design rule follows: run both arms on the same seeds. Pairing does not remove the run-to-run spread; it removes the part of the spread that the two arms share, because each per-seed difference is computed inside one seed. Section 4.1 says to keep every seed; this exercise says to keep the same seeds on both sides of a comparison. An unpaired comparison is still a comparison, and the difference it reports is the component's effect plus the difference in seed difficulty, with no way to separate the two from the table alone.