c02 E3 Open · Deeper

c02.E3.1E3 · open

You hold a table of labelled examples and a nearest-neighbour rule. You may cut the table into parts, choose $k$ , and report one number for performance on new data. The field has no procedure that guarantees the best $k$ for every data source: section 2.5 reports an experiment in which two problems built from the same features, the same sample size and the same code wanted different settings, and every estimate from a finite sample carries uncertainty that one number does not show.

Take a position on this question: what procedure should choose $k$ , and what should the reported number be allowed to promise?

In your answer: state how you split the rows and what each part is used for; say how you handle a tied vote and an even $k$ ; say what the reported number can and cannot promise about rows nobody has labelled; and name one way your procedure could mislead you even when every step is followed.

Rubric. Tick each criterion your answer meets, then save the self-assessment. The platform does not grade it.

**0 — Restatement.** The answer repeats that $k$ matters and that test data must be separate, but proposes no procedure and reports no number. Nothing is decided.**1 — A split with one job confused.** The answer names a training set and a test set, but chooses $k$ on the test set, or reports the training error, or reads the test set more than once while changing the rule. The answer does not say what the reported number promises.**2 — Three parts used correctly.** The answer uses a training set, a selection set for $k$, and a test set read once. It states a tie rule, handles an even $k$, and reports the test error rate with its denominator. It says that the number estimates performance on rows from the same source and does not promise it for a different source.**3 — Protocol defended.** The answer does everything at level 2 and also argues why its split is the right one. It distinguishes the uncertainty in choosing $k$ from the uncertainty in the final estimate, names a concrete way the procedure can mislead, such as grouped rows crossing the split or a selection set too small to separate two values of $k$, and states the assumption about new data that the number rests on. It gives the cost of its own procedure, for instance the rows removed from training or the variance introduced by a small selection set.Solution: There is no single correct answer, so this is a model of the shape a complete answer takes, not a key. A level-3 answer fixes three parts of the table before it computes anything. The training set is stored by the rule. The selection set is scored once for each candidate $k$, and the $k$ with the smallest selection error is kept; ties in selection error go to the smaller $k$, stated in advance. The test set is read once, after $k$ is fixed, and the reported number is the zero-one error rate on it, written with its denominator, such as 14 wrong in 600. The answer then says what that number can promise: it estimates the error rate on further rows drawn like the test rows, and it does not promise the same rate for rows from another writer, sensor or week. It names a failure: if rows arrive in groups, a random split can put the same group on both sides, and the test number falls while the rule has not improved. It also says that a selection set of 20 rows cannot separate two values of $k$ whose true rates differ by one percentage point, so the choice of $k$ is itself noisy. A strong answer may propose repeating the split several times and reporting the spread, but it must still say that the spread is an estimate, not a guarantee. The answer that scores 0 says only that the test set must be separate and stops there.

Open: Learning a Rule From Examples

Worked solution

Other tiers