Practise: Judging a Model Honestly

The training error is $$\frac{2}{40}=\frac{1}{20}=\mathrm{0.05.}$$ The test error is $$\frac{9}{50}=\mathrm{0.18.}$$ The optimism is the difference: $$0.18-0.05=\mathrm{0.13.}$$ The training error is 13 percentage points lower than the held-out error, which is the amount by which the model flatters itself on rows it has already seen. The sign matters: if the held-out error had been smaller, the optimism would be negative. Section 15.1 defines the gap as the price of freedom, and a report that gives only 0.05 hides that price.

The held-out fold rates are $$\frac{1}{5},\frac{3}{5},\frac{0}{5},\frac{2}{5}.$$ Their average is $$\frac{1}{4}\left(\frac{1}{5}+\frac{3}{5}+0+\frac{2}{5}\right)=\frac{1}{4}\cdot \frac{6}{5}=\frac{6}{20}=\mathrm{0.3.}$$ Adding the errors first gives the same number here because the folds are equal in size: 6 errors over 20 rows. The equality would fail with unequal folds, where the average of the fold rates is the quantity section 15.2 defines. Every row was scored exactly once and used for fitting in the other three rounds, which is what separates this estimate from a single lucky split.

Five folds hold $100/5=20$ rows each. A five-fold model trains on four folds, so it sees $$100-20=80$$ rows. The single-split model sees 70. The difference is $$80-70=10$$ rows. The extra rows are the reason cross-validation gives a less pessimistic estimate than a single split, which trains on only 70 percent of the data. The price is five fits instead of one, and the five fitted models are still not the model you will ship, because each one sees only 80 percent of the rows.

The leaky model is right on $$0.94\times 200=188$$ rows. The model without the column is right on $$0.71\times 200=142$$ rows. The gap is $$188-142=46$$ rows, which is $46/200=0.23$ of the data. Section 15.3 says the gap is not a small correction: the leaking column changes the model from one that is right about seven times in ten to one that is right more than nine times in ten. The column is not corrupt. Its values are recorded after the answer is known, so the relationship is real in the table and absent at prediction time. A clean score is therefore not evidence that the data are clean.

The resample mean is $$\frac{3+3+5+8+9}{5}=\frac{28}{5}=\mathrm{5.6.}$$ It does not have to equal 6. The original sample contains one 5 and two copies of no value, while the resample contains two 3s and no 5. Drawing with replacement changes the count of each value, and the mean moves with those counts. Here the resample mean sits $6-5.6=0.4$ below the original mean. Section 15.4 uses exactly this movement: the spread of many such replicate means estimates how much the mean would move if the experiment were run again. One resample is not an estimate of anything by itself; it is one draw from the bootstrap distribution.

The number of patterns with exactly $k$ correct is $\left(\genfrac{}{}{0ex}{}{14}{k}\right)$. For the upper tail, $$\left(\genfrac{}{}{0ex}{}{14}{11}\right)=364,\left(\genfrac{}{}{0ex}{}{14}{12}\right)=91,\left(\genfrac{}{}{0ex}{}{14}{13}\right)=14,\left(\genfrac{}{}{0ex}{}{14}{14}\right)=1.$$ The tail contains $$364+91+14+1=470$$ patterns out of $2^{14}=16384$, so $$p=\frac{470}{16384}=\frac{235}{8192}\approx 0.0286865,$$ which rounds to $0.0287$. At a significance level of 0.05 fixed before the data arrived, the filter rejects the null hypothesis, because $0.0287<0.05$. The number is the probability of a result at least this good under a coin-flipping null. It is not the probability that the null is true, and it says nothing about how many extra spam emails the filter catches in a real mailbox.

The overall error rate is $$\frac{8}{40}=0.2,$$ and the error rate in ward C is $$\frac{5}{10}=\mathrm{0.5.}$$ The ratio is $$\frac{0.5}{0.2}=\mathrm{2.5.}$$ The model is wrong two and a half times as often in ward C as it is on average, and that fact is invisible in an overall accuracy of $1-0.2=0.8$. Section 15.6 treats the gap as a lead rather than a verdict. Ward C holds only 10 patients, so one unlucky row moves its rate by 0.1, and the next question is whether the ward column is recorded consistently and whether more rows from that ward confirm the pattern.

The replicate mean is $$\frac{4.2+4.8+5.0+5.4+5.6}{5}=\frac{25.0}{5}=\mathrm{5.0.}$$ The squared deviations from 5 are $$(-0.8{)}^2=0.64,(-0.2{)}^2=0.04,0^2=0,0.4^2=0.16,0.6^2=\mathrm{0.36.}$$ Their sum is $$0.64+0.04+0+0.16+0.36=\mathrm{1.20.}$$ Divide by $B-1=4$: $$\frac{1.20}{4}=\mathrm{0.30.}$$ The standard deviation is $$\sqrt{0.30}\approx 0.5477,$$ which rounds to $0.548$. Section 15.4 uses this number as the bootstrap estimate of the standard error of the mean. Five replicates are far too few for a stable estimate in practice; the arithmetic is the same at $B=1000$, where the divisor is 999.