Derive: Lines, Curves and Boundaries

The fitted line passes through the point of averages, so its prediction is ${\hat{y}}_i=\bar{y}+b(x_i-\bar{x})$, and the residual is $$e_i=y_i-{\hat{y}}_i=(y_i-\bar{y})-b(x_i-\bar{x}).$$ Squaring and summing gives $$\mathrm{RSS}=\sum _i{e}_i^2=S_{yy}-2b{S}_{xy}+b^2{S}_{xx}.$$ Complete the square in $b$: $$\mathrm{RSS}=S_{xx}{\left(b-\frac{S_{xy}}{S_{xx}}\right)}^2+S_{yy}-\frac{S_{xy}^2}{S_{xx}}.$$ The squared term cannot be negative, so the smallest residual sum sits at $b=S_{xy}/S_{xx}$, the slope of section 14.1. Substituting that value, $$\mathrm{RSS}=S_{yy}-2\frac{S_{xy}^2}{S_{xx}}+\frac{S_{xy}^2}{S_{xx}}=S_{yy}-\frac{S_{xy}^2}{S_{xx}}.$$ Section 14.1 defines $R^2=1-\mathrm{RSS}/S_{yy}$, so $$R^2=1-\frac{S_{yy}-S_{xy}^2/S_{xx}}{S_{yy}}=\frac{S_{xy}^2}{S_{xx}{S}_{yy}}.$$ The right side is the square of the sample correlation $r=S_{xy}/\sqrt{S_{xx}{S}_{yy}}$, so for a simple fit with an intercept $R^2=r^2$. For the four points, $\bar{x}=2.5$ and $\bar{y}=3$. The products are $(-1.5)(-1)=1.5$, $(-0.5)(-2)=1$, $(0.5)(2)=1$ and $(1.5)(1)=1.5$, so $S_{xy}=5$. The input deviations give $S_{xx}=5$, and the output deviations give $S_{yy}=1+4+4+1=10$. Therefore $$R^2=\frac{5^2}{5\times 10}=\frac{25}{50}=\mathrm{0.5.}$$ The identity connects section 14.1, which defines $R^2$ from the residual and total sums of squares, with section 14.2, which reads the residuals as evidence about the model. It holds only when the model has an intercept and only for one input; with several inputs $R^2$ is not the square of any single correlation.

For the row at $x=0$ the answer is $0$, so its contribution is $1-p(0)$. For the row at $x=1$ the answer is $1$, so its contribution is $p(1)$. Independence multiplies them: $$L({\beta }_0,{\beta }_1)=(1-p(0))p(1).$$ Choose ${\beta }_0=-a$ and ${\beta }_1=2a$. The scores are $-a$ at $x=0$ and $a$ at $x=1$, so $$p(0)=\frac{1}{1+e^a},p(1)=\frac{1}{1+e^{-a}}=\frac{e^a}{1+e^a}.$$ Therefore $$L(a)=\frac{e^a}{1+e^a}\cdot \frac{e^a}{1+e^a}=\frac{e^{2a}}{(1+e^a{)}^2}.$$ At $a=0$ this is $1/4$. As $a$ grows, the numerator and denominator both grow like $e^{2a}$, and the ratio approaches $1$ from below. Every finite $a$ gives a value strictly below $1$, because both probabilities lie strictly between $0$ and $1$. No finite pair $({\beta }_0,{\beta }_1)$ attains the limit, so there is no maximizer; the likelihood has a supremum of $1$ at infinity.

This is the counterexample to the claim that a fit always has one best answer. Section 14.6 says the negative log-likelihood is convex when the mean function increases, and that is true here, but convexity guarantees a unique minimum only when the minimum is attained. Perfect separation pushes the coefficients along a direction where the loss falls forever, and the fitted probabilities approach $0$ and $1$ without ever reaching them. A table that can be separated exactly therefore has no finite logistic fit, even though every step of the arithmetic is well defined.

The fitted value at input $x$ is $ax+ax=2ax$ because the two columns are identical. Against $y=3x$ the residual is $(3-2a)x$. The residual sum of squares is $$\sum _i(3-2a{)}^2{x}_i^2=(3-2a{)}^2(1+4+9)=14(3-2a{)}^2.$$ The ridge penalty charges each coefficient separately, so it is $\lambda (a^2+a^2)=2\lambda a^2$. The total is $$F(a)=14(3-2a{)}^2+2\lambda a^2.$$ The total is a quadratic in $a$ with a positive leading coefficient, so its minimum sits at the vertex: $$a=\frac{168}{2(56+2\lambda )}=\frac{168}{112+4\lambda }=\frac{42}{28+\lambda }.$$ For general $k$ the same steps give $a=S_{xy}/(k{S}_{xx}+\lambda )$, where $S_{xy}=\sum x_i{y}_i=42$ and $S_{xx}=\sum x_i^2=14$. At $\lambda =56$, $$a=\frac{42}{28+56}=\frac{42}{84}=\mathrm{0.5.}$$ Each copy carries weight $0.5$, so the total weight is $ka=1$, and the prediction at $x=3$ is $1\times 3=3$ against the recorded $9$. At $\lambda =0$ the same formula gives $a=1.5$, an even split of the least-squares slope $3$. As $\lambda$ grows, both weights fall toward zero and the pair keeps an even split at every charge, because the table cannot tell the two columns apart. The derivation connects section 14.1, which supplies the least-squares slope through $S_{xy}$ and $S_{xx}$, with section 14.3, which adds the charge and resolves the ambiguity between identical columns.