The SVD

Take the matrix $A$$=\left(\begin{array}{cc}3& 0\\ 4& 5\end{array}\right)$ and feed it every vector on the unit circle, the set of points at distance one from the origin. The outputs form an ellipse. Its longest semi-axis is $\sqrt{45}\approx 6.71$ and its shortest is $\sqrt{5}\approx 2.24$. The matrix has turned the circle into an ellipse and stretched it by two very different amounts.

Every matrix does something of this kind, and the pattern has three steps: rotate the input, stretch along perpendicular axes, rotate the output. That is the singular value decomposition, or SVD. It writes any matrix, square or rectangular, as !The matrix $\left(\begin{array}{cc}3& 0\\ 4& 5\end{array}\right)$ turns the unit circle into an ellipse. The right singular vectors $(1,1)/\sqrt{2}$ and $(1,-1)/\sqrt{2}$ leave as $\sqrt{45}{u}_1$ and $\sqrt{5}{u}_2$, so the semi-axes are 6.71 and 2.24 and the two rotations set their directions. Status: re-simulated.

!The middle factor $\Sigma$ for a square, a tall and a wide matrix. The singular values sit on the diagonal and zeros pad the shape so that $U\Sigma V^{\top }$ multiplies; the two rotations take the remaining room. Status: schematic.

The factorization is easier to read against the ellipse the matrix has drawn above. Pick the two perpendicular input directions that the ellipse singles out: the first is stretched by $6.71$ and the second by $2.24$, and they land on perpendicular output directions. The change of axes that lines those two input directions up with the coordinate axes is $V^{\top }$. The columns of $V$ are the right singular vectors, so $V$ maps the coordinate axes onto them and $V^{\top }$ does the reverse. That is why the factorization uses $V^{\top }$ and not $V$.

The diagonal matrix $\Sigma$ then stretches along the coordinate axes by the singular values, which are never negative and are usually written in decreasing order. The matrix $U$ turns the stretched result onto the left singular vectors, the directions it actually lands on in the output space [S:27:128]. Only the middle step changes lengths: $V^{\top }$ and $U$ are orthogonal, so they preserve lengths and angles.

The decomposition exists for every matrix, including matrices with more rows than columns or more columns than rows [S:27:133]. That is a real difference from the eigenvalue story, which needs a square matrix and may still fail to supply enough eigenvectors.

The eigenvalues of $A^{\top }A$ are the squares of the singular values of $A$ [S:27:130]. Since $A^{\top }A$ is symmetric, the spectral theorem guarantees it has a full orthonormal set of eigenvectors and real eigenvalues [S:27:129]. That is where the right singular vectors come from.

Worked example. Compute

Its trace is $50$ and its determinant is $625-400=225$. An eigenvalue $\lambda$ therefore solves ${\lambda }^2-50\lambda +225=0$, which factors as $(\lambda -45)(\lambda -5)=0$. So the eigenvalues are $45$ and $5$, and the singular values are their square roots, $\sqrt{45}\approx 6.71$ and $\sqrt{5}\approx 2.24$.

The right singular vectors are the eigenvectors of $A^{\top }A$, namely $(1,1)/\sqrt{2}$ and $(1,-1)/\sqrt{2}$. Now send them through $A$. The first goes to $(3,9)/\sqrt{2}$, which has length $\sqrt{45}$ and direction $(1,3)/\sqrt{10}$. The second goes to $(3,-1)/\sqrt{2}$, which has length $\sqrt{5}$ and direction $(3,-1)/\sqrt{10}$. Those two unit directions are the left singular vectors, and they are perpendicular.

The product of the singular values is $\sqrt{45}\sqrt{5}=\sqrt{225}=15$, and the determinant of $A$ is also $15$. That agreement is not general: the product of the singular values gives $|\det A|$, the size of the determinant, and the sign is lost. The sign is $\det U\det V^{\top }$, which is $+1$ or $-1$. For a diagonal matrix with entries $-3$ and $5$ the singular values are $3$ and $5$, and the determinant carries a minus sign. So the determinant is the product of the eigenvalues, while its absolute value is the product of the singular values [S:27:118][S:24:376].

For a square matrix the two rotations have the same size. For a rectangular matrix they do not, and the middle matrix has extra rows or columns of zeros [S:27:133]. That is how one factorization covers every shape of matrix.

The number of nonzero singular values is the rank of the matrix [S:27:135]. A matrix of rank one has one nonzero singular value, and its output fills only a line. A matrix of full rank has as many nonzero singular values as its smaller side. This is why the SVD is the standard way to measure how close a matrix is to being low-rank.

A rank-one matrix is the extreme case. It sends the whole plane onto a single line, and the SVD writes it as one nonzero term, with every other term zero. Dropping the small singular values gives a lower-rank matrix that stays close to the original; that is the truncated SVD used for compression [S:27:135].

The picture has a limit. It describes what the matrix does to a circle or a sphere, not what it means for a table of data. The singular vectors of a data matrix depend on how the columns are scaled, and on whether the columns have been centred. Change either and the "important directions" change. The mathematics is exact; the interpretation is a modelling choice.

The decomposition is also not free. Computing it for a large matrix costs time and memory, and the cost grows with both dimensions. For very large matrices, practitioners use randomised or iterative versions that approximate the leading singular values rather than compute all of them. The leading ones are much cheaper to approximate than the whole list.

The formal statement. Let $A$ be a real $m\times n$ matrix of rank $r$. The SVD writes $A=U\Sigma V^{\top }$, where $U$ is $m\times m$ orthogonal, $V$ is $n\times n$ orthogonal, and $\Sigma$ is $m\times n$ with diagonal entries ${\sigma }_1\ge \cdots \ge {\sigma }_r>0$ and zeros elsewhere [S:27:125]. The number of nonzero singular values is the rank $r$ [S:27:135]. The decomposition exists for every matrix, square or rectangular.

The derivation. The matrix $A^{\top }A$ is symmetric and positive semidefinite, so the spectral theorem gives it an orthonormal eigenbasis $v_1,\dots ,v_n$ with eigenvalues ${\lambda }_i\ge 0$ [S:27:129]. Set ${\sigma }_i=\sqrt{{\lambda }_i}$ and $u_i=A{v}_i/{\sigma }_i$ for the nonzero singular values. Then $A{v}_i={\sigma }_i{u}_i$ and $A^{\top }{u}_i={\sigma }_i{v}_i$, so the two bases pair up. Stacking the vectors gives $AV=U\Sigma$, and multiplying on the right by $V^{\top }$ gives $A=U\Sigma V^{\top }$. One step is omitted: completing $U$ to a full orthonormal basis when $m>r$. That is Gram-Schmidt on any basis of the remaining directions.

The edge case. The relation ${\sigma }_i^2={\lambda }_i(A^{\top }A)$ is exact, and it is a trap when you use it as an algorithm. Forming $A^{\top }A$ squares the condition number, so a small singular value becomes a much smaller eigenvalue, and its information can fall below the precision of the arithmetic. Numerical software computes the SVD without forming $A^{\top }A$ for this reason [S:27:132]. The core said the singular values are "easy to describe in terms of eigenvalues you already know". That is true as a description and false as a recipe.

What the core hid. The core called $U$ and $V$ rotations. They are orthogonal, so they preserve lengths and angles, but an orthogonal matrix can also reverse orientation. The singular values are unique. The singular vectors are not unique when a singular value repeats, because any orthonormal basis of that singular subspace gives another valid pair. The core's example had two distinct nonzero singular values, so the ambiguity did not appear.

Where the analogy breaks. "Rotate, stretch, rotate" describes what $A$ does to a sphere. It does not describe what $A$ does to a table of data. The singular vectors of a data matrix depend on column scaling and on centring, and the theorem is silent about both. The mathematics is exact and the interpretation is a modelling choice. A second limit: the stretch factors are the semi-axes of the image of the unit sphere $S^{n-1}$ in ${\mathbb{R}}^n$, and of the solid ellipsoid when the input is the unit ball. What they do not describe is the image of a set confined to a lower-dimensional subspace, such as a circle in a plane in ${\mathbb{R}}^3$ or a cloud of points: there only the singular values of the restriction to that subspace appear.

Original sources. The historical review cited by Brunton and Kutz names the early theoretical work: Beltrami and Jordan in 1873, Sylvester in 1889, Schmidt in 1907 and Weyl in 1912 [S:52:27]. Schmidt's approximation theorem is the statement that a truncated SVD is the best low-rank approximation. Eckart and Young rediscovered it, and the result carries their names [S:52:28]. The modern statement, with the spectral-norm proof, is Theorem 4.25 in MML [S:27:137]. The same review credits Golub's computational work, which is the bridge from the theorem to the software you call today [S:52:27].

An open question. Eckart and Young fix the rank and return the best approximation at that rank. They do not say which rank to use. In a data matrix the singular values decay smoothly, and the field has not settled a selection rule that works without a downstream task. Evidence that would settle a rule: a ground-truth low-rank matrix, noise at several levels, and a measured rank error across levels the rule was not tuned on.

A frontier result. Randomised SVD computes the leading singular values without touching the full matrix: draw a random projection, sample the column space, and decompose the small projected matrix [S:52:62]. The algorithm of Halko, Martinsson and Tropp gives error bounds for the truncated decomposition [S:52:63]. It depends on the SVD twice. The target is a truncated SVD, and the guarantee is a bound in the same spectral norm that Eckart and Young minimise [S:27:137].

What to read next. Read Brunton and Kutz 1.1 for the history [S:52:27] and 1.2 for the optimality theorem [S:52:28], MML 4.5 for the construction from $A^{\top }A$ [S:27:131], and the randomised-SVD section for the large-matrix case [S:52:62]. Read the two sections together: one gives the exact optimum, the other gives the cost of reaching it.