Logistic regression

A clinic fits the probability of an event as −2 + 0.5x, where x is age in years. At x = 0 the fitted value is −2, so the model reports a probability of −2; at x = 8 it reports 2. Neither is a chance [S:49:148].

The fault lies in the form of the model rather than in the table. Any straight line fitted to answers that are only yes or no, coded 1 and 0, will at some input value dip below zero or climb past one. The model therefore eventually reports a negative chance of an event and a chance greater than certainty [S:49:148]. !The same score −2 + 0.5x read two ways. On the probability axis in (a) the line reports −2 at x = 0 and 2 at x = 8, values no chance can take; through the logistic function in (b) the same score gives 0.12, 0.50 at x = 4 and 0.88, and never leaves the unit interval. The boundary where the chance is one half is the place the score crosses zero. Status: synthetic.

The repair keeps the line and changes its destination. Instead of reading the line as a probability, read it as a score, then pass the score through a curve that can only return numbers between zero and one [S:49:148]. The curve is the logistic function, and the whole model is called logistic regression.

The symbol z is the score, still a straight line in the input, and p is the probability the model reports. The symbol e is the fixed base of the natural logarithm. A score of zero becomes a probability of one half, large positive scores approach one, and large negative scores approach zero without ever arriving [S:49:149]. Nothing about the straight line changed; only its output was bent.

The bending has a readable scale. Rearranging the equation turns the probability into a ratio of chances and then takes a logarithm:

To see the step, invert the first equation to get 1 divided by p equals 1 plus e raised to minus z; subtract one and invert again, and the ratio p divided by one minus p equals e raised to z. Taking logarithms leaves the score on the right. The ratio p divided by one minus p is the odds, and its logarithm is the log-odds [S:49:149]. That equation says the log-odds is a straight line in the input, which is why the model is still linear, on a different scale. A one-unit increase in x adds β1 to the log-odds, and therefore multiplies the odds by e raised to β1 [S:49:149].

Two readings of the same curve are worth keeping. Read from left to right, the curve is a squashing device that accepts any score at all and returns a number inside the unit interval. Read from right to left, it is a rule about evidence: equal steps along the score multiply the odds by equal factors, so the model is additive on the log-odds scale and multiplicative on the odds scale. That second reading is what makes a coefficient interpretable, because it says what one unit of the input is worth in betting terms.

With several inputs the score is a weighted sum, and each weight multiplies the odds by the same factor whatever the other inputs are. That constant multiplicative shift is a consequence of the linear score, not a separate assumption. The substantive assumption is that the log-odds is linear in the inputs with no interaction terms, so the effect of one input does not depend on another. That assumption fails when it does, and the check is to add products of inputs to the model [S:49:149] [S:110:80]. The first thing to test is therefore that assumption, especially when the fit will be used for decisions (a cut) rather than for ranking (an order with no cut).

Worked example. Take β0 = −2 and β1 = 0.5, so the score is −2 + 0.5x. At x = 0 the score is −2 and the probability is 0.12, since the exponential of 2 is about 7.4 and one over 8.4 is 0.12. At x = 4 the score is zero and the probability is exactly 0.5. At x = 8 the score is 2 and the probability is 0.88, by the same arithmetic on the other side of the curve.

Read the slope the other way. Each extra unit of x multiplies the odds by the exponential of 0.5, which is 1.65, so the odds grow by about two thirds per unit [S:49:149]. The probability grows by a different amount at every point, because the same push means more near the middle of the curve than at its ends.

The coefficients are estimated by maximum likelihood rather than by least squares, because the answers are 0 or 1. The fit chooses the coefficients that make the recorded answers most probable under a Bernoulli distribution. That means minimising the negative log-likelihood of those probabilities rather than a sum of squared Gaussian errors. Least squares could be applied to the log-odds form, but maximum likelihood is preferred for its statistical properties. The score is still a straight line; what changed is the distribution around it [S:49:149] [S:49:150].

A threshold then turns the probability into a decision. The boundary is the set of inputs at which the probability crosses that threshold, and the threshold is a choice about costs rather than a fact about the model. A clinic that must not miss a case sets the cut low and accepts false alarms, while a lender that must not refuse good customers sets it high [S:49:148]. The reported number stays the same, and only the action changes.

The idea has a cost, and the first part of it is that a straight boundary cannot wrap around an island of one class, one of the two answers the model chooses between, inside another, whatever coefficients you choose. A coefficient can also reverse sign when another input joins the model, because the new input changes what the coefficient is holding fixed; that is how students can look riskier alone and safer at equal balance [S:49:152]. The reported probability is also a statement about a group of similar cases rather than a promise about the next one. A model that says thirty per cent is vindicated by thirty cases in a hundred, and disappointed by every individual case that goes the other way. And the whole fit is only as trustworthy as the table behind it, which is the subject of the next chapter.

The formal statement. Code the answer as $y_i\in \{0,1\}$ and let $p_i$ be the probability the model reports for row $i$, with the score ${\beta }_0+{\beta }_1{x}_i$ passed through the logistic curve. The log-likelihood of the table is

$$\ell ({\beta }_0,{\beta }_1)=\sum _{i=1}^n\left[y_i\log p_i+(1-y_i)\log (1-p_i)\right].$$

The fitted coefficients maximise this total [S:49:150]. Two hypotheses come with the maximisation and L1 names neither: the rows are independent given their inputs, and the two classes overlap, so no straight line separates them perfectly.

The derivation. Differentiate the log-likelihood with respect to the coefficients. Each row contributes its residual $y_i-p_i$ times its input vector, so the gradient is a weighted sum of the inputs. The second derivative is minus a sum of terms $p_i(1-p_i)x_i{x}_i^{\mathsf{T}}$, and that matrix is positive definite whenever the inputs are not collinear and no fitted probability sits at exactly zero or one. A strictly concave function has one maximum, so the fit is unique when it exists. The weights $p_i(1-p_i)$ are largest near one half and vanish at the extremes, which is why the fit is computed by repeated weighted least squares rather than in one step: the weights themselves depend on the current fit.

The edge case. Drop the overlap hypothesis. Take one input that separates the classes: every case with the event has a larger input than every case without. Raise the slope and the likelihood keeps rising, because each row's reported probability moves toward its recorded answer and never reaches it. No finite pair maximises the total, so the maximum-likelihood estimate does not exist [S:110:81]. Software either stops with a warning or returns enormous coefficients with enormous standard errors, and both outcomes are the arithmetic reporting an honest failure. A penalty or a prior restores a finite answer, which is ridge arriving to rescue a model that has no interior optimum.

What L1 hid. L1 says the straight line changed only its output. Formally the curve also fixes the variance of each row at $p_i(1-p_i)$, and that variance sets the weight each row receives. So the link does not merely bend the answer at the end; it changes how much each row counts while the fit is computed. L1 also reads a coefficient as a change in the odds. That reading holds for the model you fitted, with its other columns held fixed [S:49:90]. Add a correlated column and the coefficient answers a different question, which is what the student reversal in L1 demonstrates.

Where the analogy breaks. A squashing device suggests decoration applied after the modelling. The curve is the inverse of the link, and the link is part of the definition of the model, on the same footing as the straight score. Remove it and you are fitting a different statistical object, not the same object with a different display.

Original sources. ISLP section 4.3 shows why a straight line fails on a yes-or-no answer, introduces the logistic curve, and derives the odds and log-odds reading [S:49:148] [S:49:149]. The same section states that the coefficients are chosen by maximum likelihood and writes the likelihood explicitly [S:49:150]. Murphy treats the same model as binomial regression, the binary case being one trial per row [S:63:446]. Molnar adds the reading practitioners use, where a coefficient's exponential is quoted as an odds ratio [S:110:80].

An open question. A reported probability is a claim that can be false even when the ranking is right. Molnar states the check plainly: ask whether sixty per cent really means sixty per cent [S:110:80]. The field has not settled how to guarantee that property for every subgroup of a table, and the fit itself gives no such guarantee. Evidence that would settle a case: reliability curves on data held out from the fit, reported separately for subgroups fixed before the analysis, rather than one averaged curve over the whole table.

A frontier method. Complete separation is the case where the maximum-likelihood fit does not exist, and the fix is now standard: add a penalty or a prior, and the coefficient becomes finite [S:110:81]. That fix is this chapter's ridge idea applied to a likelihood rather than to a sum of squares. The two sections meet here, because the penalty that stabilises a nearly collinear linear fit also stabilises a logistic fit whose classes barely overlap.

What to read next. Read ISLP section 4.3 for the failure of the straight line, the log-odds reading and the likelihood [S:49:148] [S:49:150]. Read Murphy's section 12.2 for binomial regression as a generalised-linear model [S:63:446]. Then read Molnar's treatment of logistic regression for the odds-ratio reading, the calibration check and the separation problem in one place [S:110:80].