Model criticism

A model is right on 16 of 20 patients, and the clinic is pleased. Then someone sorts the errors by ward and finds that three of the four errors sit in a single ward of five patients. The overall score was accurate and uninformative at the same time.

The section question is how to look for the way a model is wrong, and the answer is to stop reading the total and start reading the structure of the errors. Model criticism is the practice of examining where a fitted model fails, which rows it fails on, and whether the failures share a cause. It begins with the rows you already have, and it needs no new data collection. !Two residual patterns, each plotted against the fitted value. In (a) the residuals run positive at both ends and negative in the middle, a U that says a straight-line model is missing a bend in the data. In (b) the misses grow with the size of the prediction, a funnel that says one error bar cannot describe every row. The zero line is the model being right on a row. Neither shape survives in the average error, because averaging removes exactly the structure the plot is looking for. The figure is schematic. Status: schematic.

The residual is a basic tool of model criticism: it is the difference between the recorded answer and the model's answer on one row. With a single input, plot the residual against that input; with several inputs, plot it against the fitted value instead, the model's prediction for that row [S:50:104]. An ideal plot shows no discernible pattern. The presence of a pattern may indicate a problem with some aspect of the model, but it is a lead to investigate rather than proof that the model is wrong. Discreteness, dependence and the choice of axes can also produce patterns [S:50:104]. !The four wards of the worked example, five patients each, with the error rate inside each ward. The overall rate is correct and hides the one fact the clinic can act on, because the ward is where the next question begins. The data are synthetic: the twenty patients and their errors are the section's worked table. Status: synthetic.

Two patterns repay the effort of learning to see them. A U-shaped residual plot says the relationship is curved, so a straight-line model is missing a bend that the data contain [S:50:104]. A funnel-shaped plot says the spread of the errors grows with the size of the prediction, which means a single error bar cannot describe every row [S:50:107]. Neither pattern shows up in the average error, because averaging removes exactly the structure you are looking for.

The second tool is the slice, which is a partition of the rows by a feature, meaning an input column, often one the model does not use, such as ward, age band or machine. Slicing by a feature the model does use, or by the model's prediction, is also legitimate and can reveal where the model fails. Compute the error inside each slice and compare it with the overall error. A slice that is much worse than the average is a lead rather than a verdict, because a small slice produces a noisy rate and a single unlucky row can dominate it.

The third tool is the individual row, because a handful of rows can move a fit a long way. A row far from the rest of the inputs, measured on the input columns, has more room than a central row to change the fit. It becomes influential when its answer also pulls the fit, which happens when its residual is large; removing such a row can change the fitted model substantially [S:50:110]. A distant row with a small residual barely moves the fit, and an outlier close to the rest of the inputs can also have little effect [S:50:108]. If a fit depends heavily on two or three rows, then the model is reporting those rows more than the pattern, and an error in one of them can invalidate the whole result [S:50:110].

Worked example. Twenty patients are split across four wards of five, and the model's errors per ward are below.

The overall error rate and the rate in ward D are far apart.

$$\text{overall}=\frac{4}{20}=0.2,\text{ward D}=\frac{3}{5}=\mathrm{0.6.}$$

Ward D is wrong three times as often as the average, and that fact is invisible in an accuracy of 0.8. A report that gives only the total has thrown away the piece of information the clinic can act on, because the ward is where the next question begins.

A finding changes what you do next, and there are only a few honest responses. You can add the feature that the slice suggests is missing, or you can report the slice separately so that readers see the range of behaviour. A third response is to collect more rows in that slice and find out whether the difference is real. What you cannot do is quote the overall score and hope the ward never appears in the next clinic.

A slice is only as trustworthy as the column that defines it. If the ward is recorded inconsistently, the slices mix different patients together, and the difference you find may belong to the recording rather than to the model. Check the definition of the column before you believe the gap, because a miscoded feature produces a pattern that no amount of modelling will remove.

Criticism resembles reading the comments on a restaurant rather than only its star rating. The comparison breaks because comments are not a random sample of diners, and the same caution applies to slices: you can only slice by the columns you happened to record.

Slicing has a cost that grows with the number of slices. Every slice is another comparison, so a rare slice will look bad by chance. Comparing error rates across groups is not always meaningful on its own, because the groups can differ in their patients as well as in the model's behaviour, and a rate without its denominator and its uncertainty can mislead [S:109:107]. A difference between wards can also reflect a difference in patients rather than a difference in the model, which is why a pattern is a reason to look rather than a conclusion.

Criticism locates failures, and explaining them requires a separate argument about causes. A practical discipline is to choose the slices before looking, and to report how many were examined.

The formal statement. Fit a linear model by least squares. The fitted value on row $i$ is the projection of the labels onto the space spanned by the inputs, and the residual is

$$e_i=y_i-{\hat{y}}_i,$$

the recorded answer minus the fitted value [S:50:104]. The projection is described by a matrix that maps labels to fitted values, and its diagonal entry $h_{ii}$ on row $i$ is the leverage: it measures how far that row sits from the rest of the inputs. With an intercept in the model, $h_{ii}$ lies between $1/n$ and 1, and its average over the rows is $(p+1)/n$, where $p$ is the number of slope coefficients [S:50:110]. Without an intercept the lower bound is 0 and the average is $p/n$, where $p$ is the number of columns.

The derivation. The residual vector is the label vector minus its projection, so it is orthogonal to every column of the input matrix [S:63:404]. One consequence matters here. The fitted values are a combination of those columns, so the residuals are orthogonal to the fitted values: their inner product is exactly zero. If the model includes an intercept, which is a column of ones [S:63:406], the residuals also sum to zero, so their sample covariance with the fitted values is exactly zero. Without an intercept the residuals need not sum to zero, and the sample covariance need not vanish even though the inner product is zero. The residual plot can therefore show curvature or a funnel, but it can never show the linear trend that a badly wrong straight line would produce. The pattern you most want to see is the one the fitting removes by construction.

The edge case. The calculation assumes fixed inputs and errors with a common variance. When the variance grows with the fitted value, raw residuals have unequal spread even when the mean is right, and the funnel is a statement about the noise rather than about the fit [S:50:107]. When rows are dependent, as with repeated visits from one patient, the standard errors treat the rows as independent and come out too small [S:50:105].

What L1 hid. L1 treated a slice as a partition of rows by an unused column. The formal object is a comparison of conditional error rates, and a difference between slices can come from a difference in the patients rather than in the model [S:109:107]. L1 was safe because its four wards held equal numbers of rows and the gap was large. A small slice carries the noise of a small sample, and a scan over many slices produces large gaps by chance alone.

Where the analogy breaks. The restaurant-reviews analogy suggests that criticism samples the model's behaviour. A residual plot samples nothing: it displays every row you hold, on the axes you recorded. Failures inside a slice you never recorded stay invisible, and no amount of plotting recovers them.

Original sources. ISL presents the residual plot and the row-influence statistic that L1 follows [S:50:104] [S:50:110]. The slice view has a named descendant: the individual conditional expectation plot, introduced by Goldstein and colleagues in 2015. It draws one curve per row and shows how that row's prediction moves as one feature changes [S:110:228]. The corpus does not contain the paper itself, so the text cites the method through the textbook and does not quote the authors.

An open question. The field has not settled how to move from "this slice is worse" to "this slice is worse because of the model". Barocas and colleagues state that the effect of an institution on a community is an empirical and causal question that cannot be boiled down to error rates [S:109:108]. They add that a finding of error-rate disparity by itself suggests no particular intervention [S:109:108]. Evidence that would settle a given case: a design in which the slice label is assigned rather than observed, or rows matched across slices on every recorded confounder, with the error difference estimated on the matched set.

A frontier result. Conformal prediction turns a residual score into a guarantee. On a calibration set of rows the model never saw, you compute a score for each row, take a quantile, and build a prediction set that contains the answer with probability at least one minus a chosen rate [S:48:4]. The guarantee is marginal, because it averages over rows, and it can hold while every error sits in one group [S:48:16]. That is the gap between a total score and a slice, made quantitative, and group-balanced conformal prediction closes it by calibrating inside each group [S:48:16].

What to read next. Read ISL §3.3.3 for the residual plot and the row-influence statistic [S:50:104] [S:50:110], then the ICE chapter in Molnar for the per-row view and its limits under correlated inputs [S:110:228]. For the guarantee, read Angelopoulos and Bates §1 and §4.1 [S:48:4] [S:48:16].