A family of boundaries

A parcel depot counts the parcels that arrive in each hour and wants to know how that count moves with the number of vans on the road. Fit a straight line and the model will happily predict a negative number of parcels in a quiet hour. The fault sits neither in the arithmetic nor in the table, because a count cannot be negative and a straight line does not know that.

That complaint has a general answer, and the answer is a family rather than a single model. Linear regression, logistic regression and the count model all belong to one family called generalised linear models [S:63:445]. Each member is fixed by two choices. The first is a distribution for the answer, meaning a rule for how likely each possible answer is. The second is a link function that says how the mean of that distribution connects to the straight-line score [S:63:445]. !One score, −2 + 0.5x, read through the three links. At x = 0 the identity link returns −2, the log-odds link 0.12 and the log link 0.135; at x = 8 they return 2, 0.88 and 7.39. The panels agree about direction and differ about what values the answer may take. Status: synthetic.

!Both panels are logistic fits on the same two-class scatter, drawn to explain rather than measured. In (a) the score is a weighted sum of the two inputs, so the boundary where the chance is one half is a straight line, and it cuts through the island of one class. In (b) the square of the first input joins the sum, and the same logistic model draws a curve that wraps around the island: the link changes the scale on which the score is read, not the shape of the score. Status: schematic.

Here η is the link applied to the mean, μ is the mean of the answer, and the right side is the familiar straight line [S:50:181]. The mean function is the inverse of the link: it takes the score back to the mean. Choosing a link is choosing what the answer is allowed to be, and every choice carries a statement about the possible values and about the way the spread grows with the mean.

The answer fixes the pair: a yes-or-no answer takes a Bernoulli distribution, a count takes a Poisson distribution, and an unbounded continuous answer takes a Gaussian distribution [S:50:181]. Each distribution comes with its natural link. The Bernoulli takes the log-odds, so the mean is a probability squeezed between zero and one. The Poisson takes the natural logarithm, base e, so the mean is the exponential of the score and therefore positive. The Gaussian takes the identity, which leaves the mean equal to the score [S:50:181].

Each pair also fixes how the spread grows with the mean — constant for the Gaussian, p(1−p) for the Bernoulli, μ for the Poisson — and a wrong pairing shows up as residuals that fan out or pinch in. The distribution is not decoration: it states what values the answer can take and how much scatter to expect at each mean.

Worked example. Take one score, −2 + 0.5x, and read it through all three links at x = 0 and at x = 8. At x = 0 the score is −2: the linear model predicts a mean of −2, the logistic model predicts a probability of 0.12, and the count model predicts a mean of 0.135, since the exponential of −2 is about 0.14.

At x = 8 the score is 2: the linear model predicts 2, the logistic model predicts 0.88, and the count model predicts 7.39. The three answers agree about the direction of the effect and disagree about its size and its limits, which is exactly what the choice of link decides.

Reading a coefficient across links takes one extra step each time. On the identity scale a unit of x adds β1 to the answer, on the log-odds scale it multiplies the odds by the exponential of β1, and on the log scale it multiplies the mean by that same factor [S:50:180]. A coefficient of 0.5 therefore adds half a unit to a continuous answer, multiplies odds by about 1.65, and raises an expected count by about 65 per cent.

All three are fitted by maximum likelihood, and the same machinery serves each of them. The negative log-likelihood is the negative of the log of the probability the model assigns to the recorded answers. When the link is the canonical link of the chosen distribution, the score is the natural parameter. The negative log-likelihood is then convex, meaning it has a single bowl, and any local minimum is the global one.

Its second derivative is a sum of positive multiples of the input vector times its transpose, so the maximiser is unique when it exists [S:63:447] [S:63:449]. A non-canonical link such as probit or complementary log-log need not preserve convexity, and with a canonical link and classes that separate perfectly no maximiser exists at all [S:63:448] [S:110:81]. That shared machinery is why software offers them as one command with a switch for the distribution.

The section title says boundaries, and the word needs care. In a classification model the boundary is the place where the probability passes the threshold you chose, one half when false positives and false negatives cost the same. For a logistic model that place is a straight line in the inputs whatever the threshold, but moving the threshold moves the line. Products of inputs, or squares, bend the boundary, and the model stays inside the family because the score remains a weighted sum of the columns you supplied. The family does not bend boundaries by itself; it changes the scale on which a straight score is read.

The idea has a cost. A link fixes the relation between mean and spread, and the count model forces the variance to equal the mean. The Poisson distribution has one parameter, so fixing its mean fixes its variance too; real counts often refuse that demand [S:50:180]. The family also keeps the score linear in the columns you hand it, so a boundary that no weighted sum can draw stays out of reach until somebody supplies the right column.

The formal statement. A generalised linear model has three parts: a distribution for the answer, drawn from the exponential family; a linear predictor $\eta ={\beta }_0+{\beta }_{1}x$; and a link that connects the mean of the answer to that predictor. Written in canonical form, the distribution is

$$p(y\mid \eta ,{\sigma }^2)=h(y,{\sigma }^2)\exp \left(\frac{y\eta -A(\eta )}{{\sigma }^2}\right),$$

where $A$ is the log normaliser and ${\sigma }^2$ a dispersion term [S:63:445]. Two consequences follow from differentiating $A$, and they are what the link is doing:

$$\mu =A^{\prime }(\eta ),\mathrm{Var}(y)=A^{\prime \prime }(\eta ){\sigma }^2.$$

The mean is the first derivative and the variance is the second, so the choice of distribution fixes how the spread grows with the mean. That pair of facts is the formal content of L1's "every choice carries a statement about the possible values and about the way the spread grows".

The derivation. With the canonical link, so that the score is the natural parameter, the negative log-likelihood is a sum of terms $A({\eta }_n)-y_n{\eta }_n$ over the rows, up to constants. Its gradient with respect to the coefficients is a sum of residuals $y_n-{\mu }_n$ times the corresponding inputs, and its second derivative is a sum of terms $f^{\prime }({\eta }_n)x_n{x}_n^{\mathsf{T}}$, where $f$ maps the score to the mean [S:63:448]. Because $A$ is convex, $f=A^{\prime }$ increases, every term is positive semidefinite, and the Hessian is positive definite on a full-rank table, so the negative log-likelihood is convex [S:63:449]. With a non-canonical link the score is not the natural parameter, the gradient acquires an extra factor, and convexity is no longer guaranteed. Convexity gives one best answer rather than a family of them, which is why the same fitting machinery serves all three members.

The edge case. The Poisson member forces the variance to equal the mean [S:50:178]. Real counts often vary more than that, and the corpus reports the diagnostic consequence: the reported z-statistics are inflated [S:50:180]. The gradient above shows why the coefficients survive while their error bars do not. A dispersion factor multiplies the whole negative log-likelihood, so it cancels from the equation that determines the coefficients and survives in the curvature that determines the standard errors. You inherit the point estimates and lose the error bars.

What L1 hid. L1 presents the distribution and the link as two independent choices. They are not independent when you want the canonical form: the family fixes the natural parameter, and the canonical link is the one that makes that parameter the linear predictor [S:63:448]. You may choose another link, and the corpus lists alternatives, but then the score is no longer the natural parameter and the mean-variance relation is no longer read straight off $A$.

Where the analogy breaks. "One command with a switch" suggests three settings of the same output. The switch also changes the weight each row carries during the fit, because the variance function enters the curvature. Two members fitted to one table answer the same directional question with different weightings, not merely with different scales.

Original sources. ISLR section 4.6.3 states the recipe and gives the three links in a single sentence: the identity for a continuous answer, the log-odds for a yes-or-no answer, and the logarithm for a count [S:50:181]. Murphy writes the same object in exponential-family form, with the natural parameter linear in the inputs and the mean and variance obtained from the log normaliser [S:63:445]. The convexity result that makes the fit unique is his as well, and it is proved from the Hessian of the negative log-likelihood [S:63:449].

An open question. The Poisson member assumes that the variance equals the mean, and the corpus records a table where the variance sits far above the mean [S:50:178] [S:50:180]. The field has not settled a default remedy. One repair leaves the coefficients alone and rescales the standard errors. Another changes the variance function, and with it the fit. The two answers differ, and neither dominates. Evidence that would settle a case: the coverage of nominal intervals on counts with a known degree of overdispersion, comparing both remedies on the same held-out rows.

A frontier method. The family extends past the three members that L1 names. Gamma regression and negative binomial regression are generalised linear models with other distributions and therefore other variance functions [S:50:182]. The frontier here is not a new algorithm but the recognition that mean and variance travel together. Once the variance function is a modelling choice, an analyst can match it to the mechanism that produced the counts rather than to the convenience of the software.

What to read next. Read ISLR section 4.6.2 for the count model and its mean-variance assumption [S:50:178], then section 4.6.3 for the general recipe and the three links [S:50:181]. Read Murphy's chapter 12 for the exponential-family derivation and the convexity proof [S:63:445]. Then return to L1's worked example and read the same score through all three links.