Practise: Experiments That Do Not Lie

The sum is $6+7+7+9+9+11+11+12+13+15=100$, so the mean is $100/10=10.0$ errors. No individual run produced 10 errors; the mean is a summary of the ten runs and not a result that any seed delivered. The best run is 6 errors and the worst is 15, so the gap is $15-6=9$ errors. As a fraction of the 150 items the gap is $9/150=0.06$, which is 6.0 per cent. Section 4.1's ten runs have a gap of 7 errors, or 4.7 per cent; this table has a wider band, and the mean sits inside it. Report the ten seeds, the smallest value, the largest value and the band; the mean alone hides the width.

A column is a function from the 18 items to $\{0,1\}$. Each of the 18 items has 2 possible entries, and the entries are chosen independently, so the number of columns is $$2^{18}=262,144.$$ One of those columns is the label pattern itself, so a search over all columns can always find a column that matches every label, whatever the labels are. Section 4.2 uses six items, where $2^6=64$ columns suffice; eighteen items give 4,096 times as many, and twenty give 1,048,576. The count is the arithmetic reason a wide table offers less protection than its width suggests: the number of candidate columns grows exponentially with the number of items, while the split does not.

The removal effects are $$92-71=21,92-90=2,92-92=0.$$ Their sum is $21+2+0=23$. The full system scores 92, so the effects leave $92-23=69$ points unaccounted for. The components interact, and a removal effect measures what a component earns with the other two present, not a share of the whole. Component A earns its place in this system; B earns 2 points; C earns nothing that this table can separate from zero, and section 4.3 says to report that null as carefully as the win. The gap of 69 is not an error in the arithmetic. It is the part of the score that belongs to the combination rather than to any one component.

Five ablations are five full runs, and the rerun of the full system is a sixth: $$5+1=6\text{ runs}.$$ At 36 device-hours per run the sweep costs $$6\times 36=216\text{ device-hours}.$$ Section 4.3 makes the same point with Rainbow: a run of 200 million frames takes about ten days on one graphics card, so six ablations are six full runs. The cost is why an ablation is a complete retrain rather than a subtraction of weights, and why a comparison at a smaller budget is not the same experiment.

Each field left open adds a factor equal to the number of values it can take, so the number of consistent runs is the product $$6\times 4\times 3=72.$$ Section 4.4 works the same count with two fields of ten values and gets 100. A record is decisive only when the product is one, which means no field is left open. The count is a property of the record and not of the experiment: the same run can fit one record or 72, depending on what the harness wrote down.

The fractions are $$\frac{500}{800}=\frac{5}{8},\frac{150}{800}=\frac{3}{16},\frac{150}{800}=\frac{3}{16}.$$ Put them over the common denominator 16: $$\frac{10}{16}+\frac{3}{16}+\frac{3}{16}=\frac{16}{16}=1.$$ The sum is one, so every item is in exactly one part, provided the three parts are disjoint. That is the arithmetic form of the partition claim in section 4.5. A partition is necessary for a clean comparison and not sufficient: section 4.2's group leakage passes the same sum while the test items still share a group with the training items.

The full range is $50-0=50$ and the plotted range is $30-20=10$, so the magnification is $$\frac{50}{10}=5.$$ Every difference drawn on the truncated axis appears five times taller than the same difference on the full-scale axis. Section 4.6's pregnancy chart has a magnification of 200, because a 40-week range is compressed into a 0.2-week window. A magnification of 5 is smaller and still large enough to turn a small difference into a visible one. The axis is honest when the caption states the limits and the magnification, and dishonest when it leaves the reader to infer them from the bars.

The ratio is $$\frac{0.90}{0.06}=\frac{90}{6}=15.$$ The ordinary variation is 15 times the difference between the two means, so a bar chart of the two means with no spread beside them invites the reader to treat a small difference as a finding. Section 4.6 reports the pregnancy example as a gap of 0.078 weeks against a standard deviation of 2.7 weeks, which is 34.6 times. The rule is the same at both sizes: draw the spread beside the difference, and let the reader compare the two.