Eigenvalues and eigenvectors

Take the matrix $A$ below and apply it to three vectors:

!The matrix $A=\left(\begin{array}{cc}2& 1\\ 0& 1\end{array}\right)$ leaves two directions alone: it doubles $(1,0)$ and fixes $(1,-1)$. It sends $(1,1)$ to $(3,1)$, which is off both lines. The two factors, 2 and 1, are the eigenvalues. Status: re-simulated.

Applying a matrix to a vector follows one rule: take a row of the matrix, multiply it entry by entry with the vector, and add the products. That sum is the dot product of the row and the vector. For the first row and the vector $(1,0)$ it is $2\cdot 1+1\cdot 0=2$, and for the second row it is $0\cdot 1+1\cdot 0=0$. So $A(1,0)=(2,0)$. !One direction under four eigenvalues. A factor of modulus above one stretches the arrow, a factor of modulus below one shrinks it, a negative factor sends it to the opposite ray, and zero collapses it to the origin. The sign says whether the direction survives; $|\lambda |$ says by how much the length changes. Status: schematic.

The same rule gives the other two answers. It sends $(1,-1)$ to $(2-1,0-1)=(1,-1)$, which is one times itself, and $(1,1)$ to $(2+1,0+1)=(3,1)$. That last output is not a multiple of $(1,1)$: if $(3,1)=c(1,1)$, the first coordinate forces $c=3$ and the second forces $c=1$, so no single number works.

Two of the three directions survive the matrix unchanged apart from length, and the third is turned. A nonzero vector that the matrix only stretches is an eigenvector, and the stretch factor is its eigenvalue. An eigenvector $v$ and its eigenvalue $\lambda$ satisfy

The equation says that applying the matrix to $v$ gives the same result as multiplying $v$ by the single number $\lambda$ [S:24:148]. The bracketed code is a source tag: $S$ for source, then the number of the source, then the page inside it. The tag above names Axler's Linear Algebra Done Right, and tags that start with another source number name other books; the other one used throughout this chapter is Deisenroth, Faisal and Ong, Mathematics for Machine Learning.

The clause $v\ne 0$ matters, because the zero vector satisfies the equation for every $\lambda$ and carries no direction [S:23:422]. Any nonzero multiple of an eigenvector is another eigenvector for the same eigenvalue, so an eigenvalue describes a whole direction, not one arrow [S:27:111].

Read the equation one coordinate at a time. It says that $Av-\lambda v$ is the zero vector. The identity matrix $I$ has $1$ on the diagonal and $0$ everywhere else, and it leaves every vector alone, so $\lambda v=\lambda Iv$ and $(A-\lambda I)v=0$.

A square matrix sends some nonzero vector to zero exactly when it has no inverse, and the determinant is the number that is zero exactly in that case. For a two-by-two matrix with entries $a,b,c,d$ read across the rows, the determinant is $ad-bc$. So a nonzero solution exists exactly when $\det (A-\lambda I)=0$ [S:27:113].

That condition produces a polynomial, the characteristic polynomial, and its roots are the eigenvalues. For a two-by-two matrix it reads

where the trace is the sum of the diagonal entries. The two roots add to the trace and multiply to the determinant, because $(\lambda -{\lambda }_1)(\lambda -{\lambda }_2)$ expands to ${\lambda }^2-({\lambda }_1+{\lambda }_2)\lambda +{\lambda }_1{\lambda }_2$ [S:27:118][S:27:119]. The polynomial therefore carries both the sum and the product of the answer.

This is the first answer to the chapter's question. Eigenvalues matter because they describe what survives repetition. If $Av=\lambda v$, then $A^{2}v={\lambda }^{2}v$, and after repeated application the vector is a power of $\lambda$ times $v$. A direction that stays put turns a long computation into one multiplication per step. The rest of the chapter uses that fact to predict long-run behaviour, to find a dominant direction by repeated multiplication, and to decompose matrices that are not symmetric.

A diagonal matrix makes the idea visible. The matrix with $3$ and $2$ on its diagonal sends $(1,0)$ to $3(1,0)$ and $(0,1)$ to $2(0,1)$, so its eigenvalues are the diagonal entries. Nothing is turned, and nothing has to be solved.

The direction is not always visible by inspection. For the matrix

the trace is $4+3=7$ and the determinant is $4\cdot 3-2\cdot 1=10$. The same polynomial reads ${\lambda }^2-7\lambda +10=0$ [S:27:110]. Its roots are $5$ and $2$, the values MML prints for this same matrix [S:27:113-114]. Their sum is the trace and their product is the determinant. Check the directions: $(2,1)$ goes to $(10,5)=5(2,1)$, and $(1,-1)$ goes to $(2,-2)=2(1,-1)$.

The set of all eigenvectors for one eigenvalue, together with the zero vector, is a subspace called the eigenspace [S:24:178]. A subspace is a set of vectors closed under addition and under multiplication by a number, so it is a flat piece through the origin: a line, a plane, or the whole space. Here the two eigenspaces are two lines through the origin.

They are different lines, and together they fill the plane: every vector is a combination $c_1{v}_1+c_2{v}_2$ of the two, for two numbers $c_1$ and $c_2$ [S:24:179]. A combination here means the sum of scalar multiples: multiply each vector by its own number and add the results. That filling property is what the next section needs. It fails for some matrices, and the failure is worth seeing.

A negative eigenvalue sends the vector to the opposite ray and multiplies its length by $|\lambda |$. The vector shrinks when $|\lambda |<1$ and grows when $|\lambda |>1$, so the sign carries the flip and the modulus carries the size [S:27:112]. An eigenvalue of zero collapses that direction to the origin. One number therefore answers two questions at once: does the direction survive, and how fast does it grow or shrink?

The idea has a real limit. Not every real matrix has a real eigenvector. A quarter turn sends $(x,y)$ to $(-y,x)$ and turns every nonzero vector, so over the real numbers it has no eigenvalues at all [S:24:149].

Over the complex numbers, which are the reals with a square root of $-1$ adjoined, it has two [S:24:150]. They are $i$ and $-i$, the two numbers whose square is $-1$. If you assume every matrix has a real eigenvalue, this is the example that breaks it.

Finding roots of the characteristic polynomial is the textbook route, not the practical algorithm. The definition is $Av=\lambda v$ with $v\ne 0$, and $\det (A-\lambda I)=0$ is an equivalent test. Numerical software almost never forms the polynomial, because for a large matrix its roots are hard to compute and sensitive to rounding [S:28:134]. The power method, which the next section defines, takes another route: it needs only matrix-vector products, one per step, so its cost stays low when a single direction is enough.

The formal statement. Let $V$ be a finite-dimensional vector space over $\mathbb{R}$ or $\mathbb{C}$, and let $T$ be a linear operator on $V$. A scalar $\lambda$ is an eigenvalue of $T$ when $Tv=\lambda v$ for some nonzero $v$ in $V$. The eigenvectors for $\lambda$, together with the zero vector, form the eigenspace, which is the null space $\mathrm{null}(T-\lambda I)$ [S:24:178]. Because $V$ is finite-dimensional, $\lambda$ is an eigenvalue exactly when $T-\lambda I$ fails to be invertible [S:24:149], and for a matrix that failure is the condition $\det (A-\lambda I)=0$ [S:27:113].

The derivation. The equation $Tv=\lambda v$ is the same as $(T-\lambda I)v=0$. A nonzero solution exists exactly when $T-\lambda I$ has a nonzero kernel, and on a finite-dimensional space that happens exactly when $T-\lambda I$ is not invertible. Writing the operator as a matrix in any basis turns the last condition into $\det (A-\lambda I)=0$. Expanding the determinant gives a polynomial of degree $n$ in $\lambda$, so over $\mathbb{C}$ it has $n$ roots counted with multiplicity. One step is omitted: the proof that the determinant does not depend on the basis. A change of basis replaces $A$ by a similar matrix, and similar matrices have the same determinant.

The edge case. Take $A=\left(\begin{array}{cc}2& 1\\ 0& 2\end{array}\right)$. Its characteristic polynomial is $(2-\lambda {)}^2$, so $\lambda =2$ is a double root. A reader who counts roots expects two eigendirections. The equation $(A-2I)v=0$ reads $\left(\begin{array}{cc}0& 1\\ 0& 0\end{array}\right)v=0$, so the eigenspace is the single line spanned by $(1,0)$ [S:27:114]. The algebraic multiplicity is $2$ and the geometric multiplicity is $1$, and such a matrix is called defective [S:27:117]. Here the core's "one number answers two questions" breaks: the number gives the growth rate, not the number of surviving directions.

What the core hid. The core defined the eigenvalue from a matrix in the standard basis. The definition belongs to the operator, and the characteristic polynomial is the same in every basis. The core also hid the dependence on the field. Over $\mathbb{R}$ a quarter turn has no eigenvalue, because ${\lambda }^2+1=0$ has no real root [S:24:149]. Over $\mathbb{C}$ the same turn has eigenvalues $i$ and $-i$ [S:24:150]. The core said this at the end. The formal statement needs it at the start, because "the roots are the eigenvalues" is a statement about $\mathbb{C}$ unless you add a real-root condition.

Where the analogy breaks. "Directions that stay put" describes a line, not an arrow. An eigenvector with $\lambda =-1$ lands on the opposite ray, and one with $\lambda =0$ collapses to the origin. The invariant object is the line through $v$. Distinct eigenvalues always give independent eigenvectors [S:27:117], but a repeated eigenvalue may give fewer directions than its multiplicity, as the shear shows.

Original sources. The definition you read is Definition 5.5 in Axler, stated for an operator on a vector space over $\mathbb{R}$ or $\mathbb{C}$ [S:24:148]. The same definition opens Hefferon's chapter, with the older names characteristic values and characteristic vectors, and with the note that "Eigen" is German for "characteristic of" or "peculiar to" [S:23:422]. MML gives the same etymology and notes the common convention of listing eigenvalues in descending order, while stating that it does not presume an ordering unless one is given [S:27:111].

An open question. A trained network is a map whose curvature changes during training, and the largest eigenvalue of its Hessian enters the necessary condition for plain gradient descent to become unstable [S:645:10]. The field has not settled which spectral quantity of the Hessian predicts a run that will diverge rather than settle. Evidence that would settle it: measured Hessian spectra and loss curves on the same architectures, at learning rates on both sides of the threshold, with the spectra computed independently of the training code. A useful predictor must fire before the divergence, which rules out any quantity read from the loss curve after it happens.

A frontier result. The edge-of-stability picture starts from a necessary condition and adds two empirical processes. The conventional quadratic model predicts that gradient descent is unstable only if the learning rate exceeds $2/{\lambda }_{\max }$ of the Hessian: the condition is necessary, not sufficient. Progressive sharpening is the observation that while the learning rate stays below that threshold, the curvature rises until the threshold is met. Self-stabilisation is the observation that a spike then pushes the parameters into a region of smaller curvature, so the run oscillates at the edge instead of diverging [S:645:10]. That is an eigenvalue statement. It uses the same ${\lambda }_{\max }$ you met in the first section, now on a matrix that changes at every step.

What to read next. Read Chapter 6 of Solomon for algebraic and geometric multiplicity and for the Jordan form [S:28:134]. Read Chapter 5 of Axler for the operator version and for the proof that a complex operator always has an eigenvalue [S:24:148]. Then read the stability section of the small-scale-proxies paper to see the eigenvalue used as a diagnostic rather than a definition [S:645:10].