Derive: Projection and the Best Fit

(a) Write the residual list as $\mathbf{r}=\mathbf{b}-A\mathbf{c}$. Then $$S=\mathbf{r}\cdot \mathbf{r}=\mathbf{b}\cdot \mathbf{b}-2\mathbf{b}\cdot (A\mathbf{c})+(A\mathbf{c})\cdot (A\mathbf{c}).$$ The middle term is $\mathbf{b}\cdot (A\mathbf{c})=(A^{\mathsf{T}}\mathbf{b})\cdot \mathbf{c}=\mathbf{c}\cdot (A^{\mathsf{T}}\mathbf{b})$, because moving $A$ to the other side of a dot product transposes it. The last term is $(A\mathbf{c})\cdot (A\mathbf{c})={\mathbf{c}}^{\mathsf{T}}{A}^{\mathsf{T}}A\mathbf{c}$. So $$S(\mathbf{c})=\mathbf{b}\cdot \mathbf{b}-2\mathbf{c}\cdot (A^{\mathsf{T}}\mathbf{b})+{\mathbf{c}}^{\mathsf{T}}(A^{\mathsf{T}}A)\mathbf{c}.$$ (b) Section 8.2 proves that the closest point of a subspace is the one whose error is perpendicular to the subspace. The fitted list $A\mathbf{c}$ lies in the subspace spanned by the columns of $A$, and the error is $\mathbf{b}-A\mathbf{c}$. The error is perpendicular to that subspace exactly when it is perpendicular to each column. The $k$-th such dot product is the $k$-th entry of $A^{\mathsf{T}}(\mathbf{b}-A\mathbf{c})$, so the condition is $$A^{\mathsf{T}}(\mathbf{b}-A\mathbf{c})=0,\text{that is,}{A}^{\mathsf{T}}A\mathbf{c}=A^{\mathsf{T}}\mathbf{b}.$$ This is one equation per coefficient, which is why the system is square.

(c) Let ${\mathbf{c}}^{\ast }$ solve the system, so $A^{\mathsf{T}}(\mathbf{b}-A{\mathbf{c}}^{\ast })=0$. For any $\mathbf{c}$, $$\mathbf{b}-A\mathbf{c}=(\mathbf{b}-A{\mathbf{c}}^{\ast })+A({\mathbf{c}}^{\ast }-\mathbf{c}).$$ Take the squared length of both sides. The cross term is $$2(\mathbf{b}-A{\mathbf{c}}^{\ast })\cdot A({\mathbf{c}}^{\ast }-\mathbf{c})=2(A^{\mathsf{T}}(\mathbf{b}-A{\mathbf{c}}^{\ast }))\cdot ({\mathbf{c}}^{\ast }-\mathbf{c})=0,$$ because ${\mathbf{c}}^{\ast }$ solves the system. Therefore $$S(\mathbf{c})=\Vert \mathbf{b}-A{\mathbf{c}}^{\ast }{\Vert }^2+\Vert A(\mathbf{c}-{\mathbf{c}}^{\ast }){\Vert }^2=S({\mathbf{c}}^{\ast })+\Vert A(\mathbf{c}-{\mathbf{c}}^{\ast }){\Vert }^2.$$ A squared length is never negative, so $S(\mathbf{c})\ge S({\mathbf{c}}^{\ast })$, and equality holds exactly when $A(\mathbf{c}-{\mathbf{c}}^{\ast })=0$. Every solution of the normal equations is a minimiser, and two different solutions give the same fitted list $A\mathbf{c}$; that is the statement of section 8.5 that dependent columns leave the coefficients undetermined but not the fit.

(d) With one column, $A=\mathbf{a}$ is $n\times 1$ and $\mathbf{c}=c$ is a single number. The matrix $A^{\mathsf{T}}A$ is the $1\times 1$ matrix whose entry is $\mathbf{a}\cdot \mathbf{a}$, and $A^{\mathsf{T}}\mathbf{b}$ is the number $\mathbf{a}\cdot \mathbf{b}$. The normal equation is therefore $(\mathbf{a}\cdot \mathbf{a})c=\mathbf{a}\cdot \mathbf{b}$, so $$c=\frac{\mathbf{a}\cdot \mathbf{b}}{\mathbf{a}\cdot \mathbf{a}},$$ which is exactly the multiplier in the projection formula of section 8.2. The fitted values are $c\mathbf{a}$, the shadow of $\mathbf{b}$ on the line through $\mathbf{a}$.

(e) For $\mathbf{a}=(1,3)$ and $\mathbf{b}=(2,4)$, the dot products are $\mathbf{a}\cdot \mathbf{a}=1+9=10$ and $\mathbf{a}\cdot \mathbf{b}=2+12=14$, so $c=14/10=7/5$. The fitted values are $(7/5)\times (1,3)=(7/5,21/5)$, and the residual is $(2-7/5,4-21/5)=(3/5,-1/5)$. Its dot product with the column is $3/5-3/5=0$, which checks the perpendicularity condition, and the residual sum of squares is $9/25+1/25=2/5$.

(a) Each column is $(1,1)$, so each column has squared length $1^2+1^2=2$ and the two columns have dot product 2. Each column has dot product $1+3=4$ with the readings. The normal equations are $$\left(\begin{array}{cc}2& 2\\ 2& 2\end{array}\right)\left(\begin{array}{c}{c}_1\\ c_2\end{array}\right)=\left(\begin{array}{c}4\\ 4\end{array}\right),$$ and both rows say $2c_1+2c_2=4$, which is $c_1+c_2=2$.

(b) The fitted value on each row is $c_1\times 1+c_2\times 1=c_1+c_2=2$. The residual is $(1-2,3-2)=(-1,1)$ on both rows, the same for every member of the family. The residual sum of squares is $(-1{)}^2+1^2=2$, also the same. The columns are dependent, so the data fix the sum of the coefficients and say nothing about how the sum is split between them.

(c) The squared length of a solution is $c_1^2+c_2^2$. Substitute $c_2=2-c_1$: the length squared is $c_1^2+(2-c_1{)}^2=2c_1^2-4c_1+4$, a parabola with its minimum where $4c_1-4=0$, so $c_1=1$ and $c_2=1$. The shortest solution is $(1,1)$, with squared length 2. The solution $(2,0)$ gives the same fitted values $(2,2)$, the same residual $(-1,1)$ and the same residual sum of squares 2, but its squared length is 4. The two answers fit equally well and one is twice as long.

(d) The least-squares problem fixes the fitted values here and not the coefficients. "The least-squares answer" is therefore a family unless the columns are independent, and the shortest member is a convention imposed after the fit, not a consequence of the data. Section 8.5 states that convention; this example shows why it needs stating. A report that prints only $(1,1)$ hides the fact that $(2,0)$ scored the same, and a reader has no way to tell from the coefficients alone that the columns were copies.

Write the fitted line as ${\hat{y}}_i=\bar{y}+m(t_i-\bar{t})$. The residual is $$y_i-{\hat{y}}_i=(y_i-\bar{y})-m(t_i-\bar{t})=d_i-m{a}_i,$$ so the residual list is $\mathbf{d}-m\mathbf{a}$. Section 8.2's projection formula gives $m=(\mathbf{a}\cdot \mathbf{d})/(\mathbf{a}\cdot \mathbf{a})$, and with that multiplier the residual is perpendicular to $\mathbf{a}$: $(\mathbf{d}-m\mathbf{a})\cdot \mathbf{a}=\mathbf{a}\cdot \mathbf{d}-m\mathbf{a}\cdot \mathbf{a}=0$. The two pieces $\mathbf{d}-m\mathbf{a}$ and $m\mathbf{a}$ are perpendicular and add to $\mathbf{d}$, so Pythagoras gives $$\Vert \mathbf{d}{\Vert }^2=\Vert \mathbf{d}-m\mathbf{a}{\Vert }^2+\Vert m\mathbf{a}{\Vert }^2=\mathrm{RSS}+m^2\Vert \mathbf{a}{\Vert }^2.$$ Now $m^2\Vert \mathbf{a}{\Vert }^2=(\mathbf{a}\cdot \mathbf{d}{)}^2/\Vert \mathbf{a}{\Vert }^2=\Vert \mathbf{d}{\Vert }^2{\cos }^2\theta$, by the definition of the cosine in section 8.1. Substituting and rearranging gives $$\mathrm{RSS}=\Vert \mathbf{d}{\Vert }^2-\Vert \mathbf{d}{\Vert }^2{\cos }^2\theta =\Vert \mathbf{d}{\Vert }^2(1-{\cos }^2\theta ).$$ For the four rows, $\bar{t}=1.5$ and $\bar{y}=0.5$, so $\mathbf{a}=(-1.5,-0.5,0.5,1.5)$ and $\mathbf{d}=(-0.5,0.5,-0.5,0.5)$. Then $\Vert \mathbf{a}{\Vert }^2=2.25+0.25+0.25+2.25=5$, $\Vert \mathbf{d}{\Vert }^2=0.25\times 4=1$, and $\mathbf{a}\cdot \mathbf{d}=0.75-0.25-0.25+0.75=1$. The squared cosine is $1/(5\times 1)=1/5$, so the residual sum of squares is $1\times (1-1/5)=0.8$. The identity says that a small residual means the centered readings point almost along the centered times, so the cosine is near $1$ or $-1$. It does not say the line is the right model: section 8.6 fits the same four rows exactly with a cubic, and a zero residual there comes from adding columns, not from the readings falling on a line.