Fitting a line

A café with four outside tables records two quantities for every afternoon it opens: the hours of sunshine and the number of cold drinks sold, counted in tens. On four consecutive days the pairs read (1, 2), (2, 4), (3, 5) and (4, 9). The owner wants a single number for how much one sunny hour is worth in drinks, and she prefers a verdict that does not depend on the most exceptional afternoon, the fourth day with nine tens of drinks. A rule fitted to that day alone would overshoot on the other three.

That is the shape of this chapter's question, which asks how a model turns inputs into a prediction. The answer arrives in two stages: select a form for the rule, then select the numbers that occupy the form. Here the form is a straight line: each afternoon's sunshine value x is multiplied by one number and shifted by another, and the result is the predicted count, written y-hat. Call the starting level β0 and the multiplier β1; the rule is the equation below. !The four afternoons of the café table with the fitted line count = −0.5 + 2.2 × hours. The vertical segments are the residuals 0.3, 0.1, −1.1 and 0.7; they sum to zero because the line passes through the point of averages (2.5, 5), and their squares add to 1.8, the quantity the fit makes as small as it can. Status: schematic.

The equation asserts that the prediction rises by an identical amount for every additional hour, beginning from a fixed level. The number β0 is the line's intercept and β1 is its slope. A slope of 2.2 means 2.2 tens of drinks, hence 22 drinks, for each additional hour of sunshine.

Which of all possible lines is best? The usual answer is least-squares, and the criterion is expressed in terms of misses. Draw a candidate line, then measure the vertical distance between each recorded observation and the line immediately above or below it.

That distance is a residual, and the rule squares every residual and adds the squares [S:49:80]. The bracketed marks name the source page, and the chapter notes collect them. The line with the smallest total is selected, and the squaring serves two purposes: a miss above and a miss below are penalized equally, and one substantial miss is penalized more than two minor ones.

The fitted object is a linear regression, and its two coefficients have readings you can say aloud. The slope is the average increase in the output associated with a one-unit increase in the input, while the intercept is the output the line predicts when the input equals zero [S:49:81]. That plain reading explains why the method travels so far: a study of television advertising reports that an additional 1000 dollars of expenditure accompanies 47.5 additional units sold [S:49:81].

Every fitted line with an intercept also passes through the point of averages, which is (2.5, 5) in this example. The reason is the first of the two equations that define the fit: it sets the average residual to zero, and a line whose misses cancel must pass through the point of averages. So the misses above the line and the misses below it balance exactly.

Worked example. Take the four pairs and compute the slope as a ratio. For each afternoon, multiply its distance from the mean sunshine by its distance from the mean count, then add the four products. That numerator equals 11, and the sum of squared sunshine deviations in the denominator equals 5, which produces a slope of 2.2.

The intercept equals the mean count minus the slope multiplied by the mean sunshine, so 5 minus 5.5, which is −0.5. The fitted line consequently reads "count = −0.5 + 2.2 × hours".

Verify one row manually: at four hours the line predicts −0.5 plus 8.8, which is 8.3, or 83 drinks, against the 90 observed. The four residuals are 0.3, 0.1, −1.1 and 0.7, and they sum to zero, as least-squares residuals always do whenever the line contains an intercept. Squared and accumulated, they produce 1.8. The counts deviate around their mean by a total of 26 when measured the same way, so the line accounts for 1 − 1.8/26, approximately 0.931 of that variation. That proportion has a name, R², and a value close to one indicates that the line explains most of the movement [S:49:97].

One input produces a line, and two inputs produce a plane [S:49:91]. Twenty inputs produce a flat surface whose position is fixed by twenty numbers instead of two, and nothing in the recipe changes: one weight, meaning one multiplier, per input, plus an intercept, all combined. The model is cheap to state and cheap to evaluate: one weight per input plus an intercept, and one multiplication and one addition per prediction. That cost is the reason to try it first on an unfamiliar table.

The idea has a price. A line can predict impossible values, and at zero hours of sun this fitted model says −0.5 tens, which is minus five drinks. A fit says nothing reliable about inputs outside the range you measured, and the further you travel from that range, the worse the guess becomes. Real relationships are seldom straight, so the linear form is a working model rather than a belief [S:49:82].

The formal statement. Collect the afternoons into pairs $(x_i,y_i)$ and write a candidate line as ${\hat{y}}_i={\beta }_0+{\beta }_1{x}_i$. The residual sum of squares is

$$\mathrm{RSS}({\beta }_0,{\beta }_1)=\sum _{i=1}^n{\left(y_i-{\beta }_0-{\beta }_1{x}_i\right)}^2.$$

Least squares returns the pair that minimises this total [S:49:80]. The minimiser exists and is unique under two hypotheses: $n\ge 2$, and the inputs are not all equal. No hypothesis mentions a straight-line truth, and none mentions Gaussian errors.

The derivation. Differentiate the total with respect to ${\beta }_0$ and ${\beta }_1$, then set both derivatives to zero. The first equation says the residuals sum to zero, and it gives ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$. Substituting that into the second equation leaves

$${\hat{\beta }}_1=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}.$$

The café table puts 11 in the numerator and 5 in the denominator, so the slope is 2.2 and the intercept is −0.5. The denominator is positive exactly when the inputs differ, which is what makes the ratio meaningful.

The edge case. Set every sunshine value equal and the denominator is zero: the slope does not exist, because every line through the average of the counts fits equally well. Drop the intercept and three familiar facts fail together. The residuals need not sum to zero, the line need not pass through the point of averages, and R² can turn negative. L1 uses the zero sum as a check on the arithmetic. It is a check rather than a test, because the first normal equation forces that sum for every table you could hand it, including a table with no relationship at all.

What L1 hid. L1 described the answer as the line with the smallest total. Formally the fitted values are the orthogonal projection of the response vector onto the plane spanned by a constant column and the input column. The two normal equations are the two orthogonality conditions, so the residual vector is orthogonal to both columns. The coefficient vector is unique whenever the two columns are independent, and the café table satisfies the condition. The simplification was safe because the geometry and the arithmetic agree on this table.

Where the analogy breaks. L1 reads the slope as what an hour of sunshine is worth. The fitted number is an association inside a chosen model, and the corpus says "associated with" rather than "causes" [S:49:81]. Add a column and the same slope can move, because the model then answers a different question. The "worth" reading survives only if the table holds every common cause, which is a claim about the world rather than about the fit.

Original sources. The corpus presents least squares as the standard criterion, defines the residual, and gives both closed forms, noting that "some calculus" produces them [S:49:80]. It then reads the two coefficients as an expected value at the input of zero and an average increase per unit of input [S:49:81]. Murphy writes the same model as a Gaussian conditional distribution, with the mean linear in the inputs [S:63:445]. The corpus does not name an inventor for the criterion, so this chapter should not either.

An open question. A fitted line is easy to state and an honest interval around its prediction is not. Split conformal prediction turns any fitted model into a prediction set with a coverage guarantee, and the guarantee rests on one hypothesis that cannot be dropped: the calibration pairs and the test pair are independent draws from the same distribution. Under that exchangeability, the probability that the set contains the test answer is at least 1 − α whatever the model, right or wrong, and at most 1 − α + 1/(n+1) when the score is continuous [S:48:6] [S:48:4]. The model and the shape of the distribution are unrestricted; their being shared is not. The guarantee averages over calibration and test points. Conditional coverage, which promises the same rate at every input value, is impossible to achieve in the general case [S:48:13]. Evidence that would settle a given case: a finite-sample lower bound on coverage within each of a list of subgroups fixed before the analysis, measured on data held out from both the fit and the calibration.

A frontier method. Conformalised quantile regression combines a quantile fit with a conformal correction. Quantile regression generalises least squares by replacing the squared loss with a pinball loss, which targets a conditional quantile rather than the mean [S:48:9]. The conformal step shifts both fitted quantiles by the calibration quantile q̂, so the interval becomes [t̂(α/2)(x) − q̂, t̂(1−α/2)(x) + q̂], and that widened band satisfies the coverage property on fresh data [S:48:8]. The method depends on this section's object twice: the fitted line supplies the centre of the band, and the residuals supply the scores that set its width.

What to read next. Read ISLP section 3.1 for the criterion and both formulas [S:49:80], then section 3.1.3 for R² and the residual standard error [S:49:97]. Read Murphy's chapter 12 for the Gaussian member of the generalised-linear family [S:63:445]. Then read Angelopoulos and Bates, sections 1 and 2, for the coverage guarantee and the quantile construction [S:48:4] [S:48:9].