The shadow of a vector

Hold a pencil above a table under a lamp. The shadow is a shorter pencil lying flat on the table, and the tip of the real pencil sits directly above the tip of the shadow. Jim Hefferon's linear algebra text uses this picture for projection and adds a warning that some shadows fall upward [S:23:285]. An upward shadow is still a shadow, and the rule that produces it does not care which side of the table the pencil occupies.

The shadow answers a question with a measurement attached: among all the points of the table, which one lies closest to the tip? The answer is the foot of the perpendicular, and no other point of the table ties with it [S:25:139]. The arrow from the shadow to the tip is the error of the approximation, and it meets the table at a right angle [S:24:202]. A projection is that closest point. The flat surface you project onto is a subspace, meaning a sheet through the origin such as a line, a plane or a higher-dimensional relative of the two. !The list b=(3,4) projected onto the line through s=(1,2) lands at (2.2,4.4); the error (0.8,−0.4) meets the line at a right angle and is 0.894 long. Right: any other point of the line, here u=2s, adds a step perpendicular to the error, so its squared distance is 0.8+0.2=1, strictly larger. Status: re-simulated.

For a line the formula is short. Write the direction of the line as $\mathbf{s}$ and the list you are projecting as $\mathbf{b}$. The closest point is some multiple of the direction, and the multiplier is the dot product of the two lists divided by the dot product of the direction with itself. The operation that produces it is written ${\mathrm{proj}}_{\mathbf{s}}$.

The formula says that the shadow keeps only the component of the list that points along the line, and it discards every other component. The discarded part is perpendicular to the line, which is exactly the condition the shadow must satisfy. The same recipe works for a table with a thousand entries, where the surface becomes a subspace you cannot picture. The perpendicular condition then becomes one equation for each direction the subspace contains: if the subspace has directions ${\mathbf{s}}_1$ through ${\mathbf{s}}_k$, the error must have zero dot product with each of them, so $k$ equations pin down the shadow. Nothing about the argument depends on the number of entries, only on the fact that a perpendicular can be defined and measured.

Worked example. Project the list (3, 4) onto the line through (1, 2). The dot product of the two lists is 3 + 8 = 11, and the direction with itself gives 1 + 4 = 5, so the multiplier is 11/5 = 2.2. The shadow is 2.2 × (1, 2) = (2.2, 4.4), and the error is (0.8, −0.4).

That error is perpendicular to the direction, because 0.8 × 1 + (−0.4) × 2 = 0, and its length is the square root of 0.8, or about 0.894. So (2.2, 4.4) is the closest point of that line to (3, 4), and the distance from the point to the line is 0.894.

No other point of the line can beat the shadow, and the reason is Pythagoras. Take any other point of the line and call it $\mathbf{u}$. The error from the list to $\mathbf{u}$ splits into the error to the shadow plus the step from the shadow to $\mathbf{u}$, and those two pieces are perpendicular. Its squared length is therefore the squared distance to the shadow plus a positive square, which is strictly larger [S:25:139]. Minimising distance and satisfying the perpendicular condition are the same problem, and that equivalence is what makes projection computable rather than a search.

Notice what the projection did not do. It is not the opening picture, which shortens a pencil onto the whole table; the projection here took a list already lying in the page and dropped a perpendicular onto a line drawn on the page. The first operation projects onto the whole plane, the second onto a line inside it, and the two coincide only when the line is the whole plane. That happens in one dimension, where the shadow of a point on a line is the point itself and the error is zero, which is a useful edge case to keep in mind when a formula returns a multiplier of 1.

The idea breaks when distance is not the thing you care about. Projection weights every direction equally, so a model that minimises it treats an error of one unit in one column exactly like an error of one unit in another, whatever their units and whatever their importance. It is also a one-way operation, because the shadow alone cannot recover the pencil; the perpendicular component is gone for good. And the answer is constrained to lie in the subspace, so a subspace that cannot contain a good answer returns the least bad point in it with no warning at all. The next section asks how to choose a basis that makes this calculation cheap and stable.

The formal statement. Let $E$ be a finite-dimensional subspace of an inner product space, and let $\mathbf{b}$ be a list. The orthogonal projection of $\mathbf{b}$ onto $E$ is the list $p$ in $E$ with $\mathbf{b}-p$ orthogonal to every list in $E$ [S:25:139]. The projection theorem makes three claims: $p$ exists, $p$ is unique, and $p$ is the closest point of $E$ to $\mathbf{b}$.

The proof. Choose an orthonormal basis $q_1,\dots ,q_r$ of $E$ and set $p={\sum }_k(\mathbf{b}\cdot q_k)q_k$. For each $j$, the dot product of $\mathbf{b}-p$ with $q_j$ is $\mathbf{b}\cdot q_j-\mathbf{b}\cdot q_j=0$. The error is therefore orthogonal to every basis list and, by linearity, to every list in $E$. That proves existence.

For uniqueness, suppose $p$ and $p^{\prime }$ both work. Then $p-p^{\prime }$ lies in $E$ and is orthogonal to $E$, so its squared norm is zero, and $p=p^{\prime }$.

For optimality, take any $x$ in $E$. Write $\mathbf{b}-x=(\mathbf{b}-p)+(p-x)$. The first term is orthogonal to $E$ and the second lies in $E$, so Pythagoras gives

$$\Vert \mathbf{b}-x{\Vert }^2=\Vert \mathbf{b}-p{\Vert }^2+\Vert p-x{\Vert }^2.$$

The second term is never negative, so $p$ wins, and it ties only when $x=p$ [S:25:140]. Nothing is omitted here except the construction of the orthonormal basis, which is Section 8.3.

The edge case. The theorem needs $E$ to be a subspace. Drop that and both claims fail. Project onto a circle in the plane instead of a line. A point at the centre of the circle ties with every point of the circle, so the nearest point is not unique, and it is not found by a formula that assumes a linear surface. L1 said "no other point of the table ties with it". That is true for a subspace and false for a curved set, and the difference is linearity, not dimension. Finite dimension matters for existence: an infinite-dimensional subspace need not contain a closest point.

What L1 hid. L1 called $p$ "the closest point" before showing that a closest point exists. In finite dimensions existence is free, because the orthonormal basis supplies the formula. L1 also said the projection "discards the perpendicular part". The proof shows why that is forced: the perpendicular part is exactly what $E$ cannot represent, and subtracting more or less of it increases the error. The simplification was safe because the worked example projected onto a line in two dimensions, where the foot is visible and unique.

Where the analogy breaks. A shadow is cast by light travelling in one fixed direction, which is an oblique projection, not an orthogonal one. An oblique shadow is not the closest point, and its error is not perpendicular to the surface. The pencil picture draws the orthogonal case; it does not define projection.

Original sources. Treil states the projection as a definition with two conditions and proves that it is the unique minimiser [S:25:139]. Axler derives the same object as an orthogonal decomposition of one list along another [S:24:202]. Hefferon gives the line formula with the shadow picture and warns that some shadows fall upward [S:23:285]. The warning matters: the picture is a guide to the orthogonal case, not a definition of projection in general.

An open question. The theorem is exact once a distance is fixed, and the field has not settled how to fix it. Weighted least squares replaces the sum of squared errors with a weighted sum and lets the weights encode how much each row matters [S:21:249]. Nothing in the geometry says which weights are right, and two weightings can rank two models differently. Evidence that would settle a given case: held-out errors under both weightings, measured against a stated cost for each kind of mistake, rather than a default preference for equal weights.

A frontier method. The Johnson–Lindenstrauss lemma projects data onto a random subspace of dimension proportional to the logarithm of the number of points, and the projection preserves all pairwise distances within a stated tolerance [S:20:159]. It depends on this section's object twice, because the map is a scaled orthogonal projection onto a random subspace and the proof compares lengths before and after it. With $m\ge C{\epsilon }^{-2}\log N$ and $Q=\sqrt{n/m}P$, every pairwise distance changes by a factor between $1-\epsilon$ and $1+\epsilon$ [S:20:160]. The scaling is what makes the map an approximate isometry: $P$ on its own shortens a typical distance by about $\sqrt{m/n}$, so it contracts rather than preserves. The result is why random projection works as compression rather than as a guess.

What to read next. Read Treil's section on orthogonal projection [S:25:139] for the proof and the uniqueness argument, Axler's section 6A [S:24:202] for the decomposition, and Boyd's chapter 12 [S:21:235] for least squares as the same problem written in coordinates. Then read Vershynin's treatment of the Johnson–Lindenstrauss lemma [S:20:159] for what survives a projection to low dimension, and Boyd's weighted least squares exercise [S:21:249] for the distance you are choosing when you weight rows.