The spectral theorem

Take the matrix $A=\left(\begin{array}{cc}2& 1\\ 1& 2\end{array}\right)$ and two vectors. It sends $(1,1)$ to $(3,3)$, which is three times $(1,1)$. It sends $(1,-1)$ to $(1,-1)$, which is one times itself. So the special directions are the diagonal and the anti-diagonal, and they meet at a right angle.

That perpendicularity is the interesting part. For a general matrix the eigenvectors can point almost anywhere, and two of them can be nearly parallel. For a symmetric matrix, one that equals its own transpose, the eigenvectors can always be chosen perpendicular, and the eigenvalues are always real [S:27:117]. The transpose swaps rows and columns, so the entry in row $i$ and column $j$ becomes the entry in row $j$ and column $i$; a symmetric matrix is one this swap leaves unchanged. This is the spectral theorem, and it is the reason symmetric matrices are the well-behaved ones. !The angle between eigendirections is a gift of symmetry. For $\left(\begin{array}{cc}2& 1\\ 1& 2\end{array}\right)$ the two lines meet at a right angle; for $\left(\begin{array}{cc}1& 1\\ 0& 2\end{array}\right)$ they meet at 45 degrees; for a quarter turn there is no real eigendirection to draw. Status: re-simulated.

The theorem says more than perpendicularity. A real symmetric matrix has an orthonormal basis of eigenvectors [S:27:117]. Unit length plus perpendicular is what orthonormal means. In that basis the matrix is diagonal: it stretches along each axis by the corresponding eigenvalue and does nothing else. Written in the original coordinates, the matrix is !$Q^{\top }AQ$ read as three moves. The axes rotate onto the eigenvectors, the plane stretches along each by its eigenvalue, and the axes rotate back. The circle becomes an ellipse, and the right angle between the eigendirections survives all three moves. Status: schematic.

where $D$ is diagonal and $Q$ is an orthogonal matrix, a matrix whose columns are orthonormal [S:27:117].

Multiplying by $Q^{\top }$ changes coordinates to the eigenvector axes, and multiplying by $Q$ changes back. The dot product of two vectors is the sum of the entry-by-entry products, the same rule that applied a matrix at the start of the chapter. Because the columns of $Q$ are unit eigenvectors, $Q^{\top }v$ lists the dot products of $v$ with each of them, which are exactly the coordinates of $v$ on those axes. For a two-by-two matrix, $Q$ is a rotation or a reflection. That is the exact sense in which a symmetric matrix can be rotated into a diagonal one.

Symmetry has a geometric meaning. It says the matrix treats the two coordinate axes alike: the number that sends the first axis into the second equals the number that sends the second into the first. The spectral theorem turns that promise into perpendicular axes and real stretch factors.

Read the three factors in order. First $Q^{\top }$ rotates the input axes onto the eigenvectors. Then $D$ stretches along those axes. Finally $Q$ rotates the result back. Each step is either a rigid motion or a pure stretch, and neither one mixes the axes.

Worked example. For the matrix above, the unit eigenvectors are $v_1=(1,1)/\sqrt{2}$ and $v_2=(1,-1)/\sqrt{2}$, with eigenvalues $3$ and $1$. Put them in the columns of

The columns have length one and their dot product is $1-1=0$, so $Q$ is orthogonal. Multiply $Q$ by its transpose and the identity comes out, which is the test. Now compute $AQ$. The first column becomes $3v_1$ and the second becomes $1v_2$, so

Multiplying on the left by $Q^{\top }$ gives

The off-diagonal entries cancel because the eigenvectors are perpendicular. That cancellation is the theorem in one calculation. The two diagonal entries are the eigenvalues, and the zeros off the diagonal say that the axes do not interact.

The same statement has a useful reading in reverse. A symmetric matrix is a sum of rank-one pieces, one per eigenvector, each weighted by its eigenvalue:

A rank-one matrix is one whose output fills a single line, and rank is defined later in the chapter. The piece $v_1{v}_1^{\top }$ is the outer product of $v_1$ with itself: it takes a vector $x$, computes the number $v_1^{\top }x$, and returns that number times $v_1$. Each piece stretches along one perpendicular direction.

For a real symmetric matrix the stretch along the unit eigenvector $v_i$ is $|{\lambda }_i|$, so the largest stretch is the largest modulus, which is the largest singular value and the spectral norm [S:27:137][S:27:130]. The largest signed eigenvalue is a different quantity: it is the largest value of $x^{\top }Ax$ on the unit circle, and it sits below the largest stretch whenever the negative eigenvalue has the larger modulus. If one eigenvalue is much larger in modulus than the other, one piece carries almost all of the action, and the matrix is close to a rank-one map.

This is why the spectral theorem sits underneath principal component analysis, which keeps the few directions with the largest stretch, and vibration modes, which are the standing-wave shapes of a structure. The next section applies the same decomposition to quadratic bowls.

The theorem is a statement about symmetric matrices, and it stops there. A general real matrix can have complex eigenvalues, and its eigenvectors need not be perpendicular. A quarter turn is the smallest example: no real eigenvector at all [S:24:149]. Even among matrices with real eigenvalues, perpendicularity is not promised without symmetry.

Without symmetry, perpendicularity can fail. The matrix $\left(\begin{array}{cc}1& 1\\ 0& 2\end{array}\right)$ has eigenvectors $(1,0)$ and $(1,1)$ for eigenvalues $1$ and $2$. Their dot product is $1$. So the angle between them is $45$ degrees, not $90$.

Repeated eigenvalues add a practical wrinkle. The theorem guarantees that an orthonormal eigenbasis exists, but the first basis you compute for a repeated eigenvalue may not be orthogonal. Gram-Schmidt is the procedure that takes a list of independent vectors and replaces it, one vector at a time, by perpendicular unit vectors pointing the same way, and MML does exactly that for a repeated eigenvalue [S:27:118]. The theorem also does not make the computation free. Finding $Q$ is an eigenproblem, and for a large matrix it costs real time and can be sensitive to rounding.

The formal statement. Let $V$ be a finite-dimensional real inner product space and $T$ a linear operator on $V$. The real spectral theorem says that three conditions are equivalent: $T$ is self-adjoint, $T$ has a diagonal matrix with respect to some orthonormal basis, and $V$ has an orthonormal basis of eigenvectors of $T$ [S:24:259]. For an operator, self-adjoint means $T=T^{\ast }$. For a real matrix this reads $A=A^{\top }$ when, and only when, $A$ is the matrix of $T$ with respect to an orthonormal basis, because that is the basis in which the matrix of $T^{\ast }$ is the transpose [S:24:259]. The eigenvalues are real, and eigenvectors for distinct eigenvalues are orthogonal [S:28:133].

The derivation. One direction is short. If $A=QD{Q}^{\top }$ with $Q$ orthogonal and $D$ diagonal, then $A^{\top }=Q{D}^{\top }{Q}^{\top }=QD{Q}^{\top }=A$, so $A$ is symmetric. The other direction is the hard one. Axler proves it through the minimal polynomial. He shows that the minimal polynomial of a self-adjoint operator has distinct real roots, using the lemma that $T^2+bT+cI$ is invertible when $b^2<4c$ [S:24:257]. That factorization gives an upper-triangular matrix in an orthonormal basis. A symmetric upper-triangular matrix is diagonal, which completes the proof [S:24:259]. I omit the proof of the upper-triangular step. It is Schur's theorem and it needs the minimal polynomial. The omitted step is named, not hidden.

The edge case. Repeated eigenvalues break the uniqueness of $Q$, not the theorem. In the three-by-three example with eigenvalues $1$, $1$ and $7$, the first two computed eigenvectors are not orthogonal, and Gram-Schmidt repairs them without leaving the eigenspace [S:27:118]. A reader who computes eigenvectors one at a time and stops at the first basis gets a $Q$ that is not orthogonal. The theorem promises that some orthonormal basis exists. It does not promise that your solver returns one.

What the core hid. The core said that $Q$ is "a rotation or a reflection". That is right in two dimensions, where every orthogonal matrix has determinant $+1$ or $-1$. In higher dimensions an orthogonal matrix is a composition of rotations in several planes, possibly with a reflection. The phrase "rotated into a diagonal one" is therefore a two-dimensional picture of a statement that holds in every dimension. It was safe in the core because the worked example was two-by-two.

Where the analogy breaks. The theorem is about symmetric matrices and stops there. Over the complex numbers the right class is the normal matrices, those with $A{A}^{\ast }=A^{\ast }A$, and the complex spectral theorem gives a unitary diagonalisation for exactly those [S:24:257]. A general real matrix can have complex eigenvalues and eigenvectors that are not perpendicular. The matrix $\left(\begin{array}{cc}1& 1\\ 0& 2\end{array}\right)$ from the core is the smallest real example. Its eigenvalues are $1$ and $2$, and its eigenvectors meet at $45$ degrees.

Original sources. The real spectral theorem is Theorem 7.29 in Axler, stated as three equivalent conditions [S:24:259]. The matrix form $A=QD{Q}^{\top }$ with an orthonormal $Q$ is the version in MML [S:27:117]. The orthogonality of eigenvectors for distinct eigenvalues is Proposition 6.4 in Solomon, with a two-line inner-product proof [S:28:133]. Over the complex numbers the same theorem replaces symmetric by normal and orthogonal by unitary [S:24:257]. Axler proves the complex version separately, because the real proof needs a lemma that the complex proof does not use [S:24:257].

An open question. The theorem says that an orthonormal eigenbasis exists. It says nothing about how accurately a computed eigenvector points when two eigenvalues are close. The field has not settled a practical rule for reporting that error in a data analysis, where the covariance matrix is itself estimated from a finite sample. Evidence that would settle it: a bound computed from the eigenvalue gap and the sample size, tested against eigenvectors from data with a known generating covariance.

A frontier result. Principal component analysis is the spectral theorem applied to a data covariance matrix. The principal subspace is spanned by the eigenvectors with the largest eigenvalues [S:27:338]. That is why one decomposition serves compression, denoising and exploratory plots. It also gives the rank-one expansion: a symmetric matrix is a sum of pieces, one per eigenvector, each weighted by its eigenvalue. The theorem fixes the algebra and leaves the modelling choices, such as centring and scaling, to you. Rescale one column and the leading eigenvector moves, although the theorem still holds.

What to read next. Read Axler's spectral theorem section for the full proof [S:24:259], Solomon 6.2 for the short orthogonality proof [S:28:133], and MML 10.4 for the step from an eigenvector to a principal component [S:27:338].