A team fits many candidate models, chooses one by cross-validated error, then reports that same cross-validated error with a bootstrap interval as the model's expected error on new patients. The field has no agreed procedure for this report. Section 15.2 says cross-validation estimates a procedure rather than the one model that was chosen. Section 15.3 says a test set stops being a test set once decisions are made with it. Section 15.4 says the bootstrap reuses one sample and cannot invent rows that the sample never contained.
Take a position: what protocol should produce the headline score, and what may the reported interval promise?
In your answer: state which rows choose the model and which rows score it; say how you compute the interval and what it is an interval for; say what the interval covers and what it does not cover; and name one way the protocol could mislead even when every step is followed.
Rubric. Tick each criterion your answer meets, then save the self-assessment. The platform does not grade it.
Worked solution
There is no single correct answer, so this is a model of the shape a complete answer takes, not a key.
A complete answer begins by fixing the target of the report. The narrow target is the expected error of the chosen fitted model on new rows from the same population. The wider target is the expected error of the whole selection procedure, including the fact that the procedure chose this model out of many. These are different quantities, and a level-3 answer says which one it is reporting.
The protocol then separates selection from reporting. One option is to split the rows into a selection part and a report part: cross-validation or a validation split on the selection part chooses the model and its settings, the model is fitted on the selection part, and the report part is scored once. The report part has never been used for any decision, so its error is an honest estimate for the chosen model. A second option is nested cross-validation: an inner loop chooses the model inside each outer training fold, and the outer fold scores the whole selection procedure. The first option estimates the chosen model; the second estimates the procedure. The answer names which one it uses and why.
The interval comes next. On the report part the team can use a binomial standard error for an error rate, or a bootstrap over the report rows, or a paired comparison if two models are being compared. The interval is computed after the model is fixed, and it covers the sampling variability of the report set under the stated model. A level-3 answer says what the interval does not cover: it does not cover a change in the population, a leaking column that survived the split, dependence between rows such as repeated visits from one patient, or the possibility that the report set is too small to separate two models whose true errors differ by little.
The cost belongs in the answer. The report part is removed from selection, so the chosen model is fitted on fewer rows and the interval is wider than it would be with all the data. The selection part is noisy, so the chosen model may not be the best one, and the report cannot show that loss because it only sees the winner. Repeating the split several times and reporting the spread of the whole protocol is one response, but the spread is still an estimate, not a guarantee.
The named failure should be concrete. A leaking column that is present in both parts inflates the report score and the interval will not reveal it, because the interval only measures row-to-row variability. A report set drawn from a different hospital than the deployment hospital makes the interval describe the wrong population. A selection set so small that two models tie makes the choice close to random, and the report then certifies a model that was chosen by noise. A level-3 answer chooses one of these, says how it would show up, and says what check would catch it.
Rubric: - 0 — Restatement. The answer repeats that training error is optimistic and that a test set must be held out, but names no protocol, computes no interval and promises nothing. No row is assigned a job. - 1 — One number, one confusion. The answer reports the cross-validated error of the winning model as if cross-validation had certified that model, or computes a bootstrap interval on the same rows that chose the winner, or says the interval is "95 percent likely to contain the true error". It does not separate selection from reporting or say what population the promise is about. - 2 — Selection separated from reporting. The answer holds out a report set, or uses nested cross-validation, so that the rows which choose the model are not the rows which score it. It states the target quantity, computes an interval from the report rows, and gives a coverage claim about repeated samples from the stated population. It names at least one thing the interval does not cover, such as a population change, leakage or dependence between rows. - 3 — Protocol defended. The answer does everything at level 2 and also argues why its split or nesting matches the target it claims. It distinguishes the uncertainty in choosing the model from the uncertainty in the final score, gives the finite-sample cost of the choice, such as rows removed from selection or the width added by a small report set, and explains how the report changes as the sample grows. It names a concrete failure that the interval cannot detect, states the assumption about the population that the whole protocol rests on, and says what evidence would make it abandon that assumption.