Eigenvalues and Eigenvectors

Vectors that keep their direction

This is the first lecture on eigenvalues and eigenvectors — “a big subject that will take up most of the rest of the course.” The matrices are square again, and we’re hunting for some special numbers and some special vectors.

What does a matrix do? It acts on vectors: in goes $\mathbf{x}$, out comes $A\mathbf{x}$. It’s like a function in calculus — in goes a number $x$, out comes $f(x)$ — except we’re up in more dimensions. The vectors Strang is specially interested in are the ones that come out in the same direction they went in. That won’t be typical: for most vectors, $A\mathbf{x}$ points somewhere new. But certain vectors come out parallel to themselves, and those are the eigenvectors.

Take a second on $\lambda = 0$. If the eigenvalue is zero, then $A\mathbf{x} = 0\mathbf{x} = \mathbf{0}$ — the eigenvectors with eigenvalue zero are exactly the vectors in the null space. So if $A$ is singular, meaning it takes some nonzero vector into zero, then $\lambda = 0$ is an eigenvalue. From the eigenvalue point of view, zero is no special deal anymore — it’s just one of the eigenvalues.

Notice the difficulty: this is not $A\mathbf{x} = \mathbf{b}$. We can’t just run elimination, because there are two unknowns, $\lambda$ and $\mathbf{x}$, and they’re multiplied together. We’ll need a good idea. But first, some matrices where we can see the answer without any formulas.

Two matrices we can read by eye

A projection matrix. Let $P$ project every vector onto a plane. Take some vector $\mathbf{b}$ poking out of the plane — is it an eigenvector? No: its projection $P\mathbf{b}$ points in a different direction. So which vectors do get projected onto themselves?

Any vector already in the plane: project it and you get it back, unchanged. So every $\mathbf{x}$ in the plane is an eigenvector, and comparing $P\mathbf{x} = \mathbf{x}$ with $A\mathbf{x} = \lambda\mathbf{x}$, the eigenvalue is $\lambda = 1$. That’s a whole plane of eigenvectors — two independent ones. In three dimensions we hope for three, and the third is the vector perpendicular to the plane: its projection is zero, so $P\mathbf{x} = \mathbf{0} = 0\mathbf{x}$, and the eigenvalue is $\lambda = 0$. The eigenvalues of a projection matrix: one and zero — read off the picture, no formula needed.

A permutation matrix. Now take

$$A = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix},$$

which switches the two components of $\mathbf{x}$. What vector is unchanged when you swap its components? $\mathbf{x}_1 = (1, 1)$: then $A\mathbf{x}_1 = (1,1) = \mathbf{x}_1$, so $\lambda_1 = 1$. A two-by-two matrix should have a second eigenvalue, and this one is going to be negative even though the matrix has no negative entries. Send in $\mathbf{x}_2 = (-1, 1)$: swapping gives $(1, -1)$, which is minus the original. So $A\mathbf{x}_2 = -\mathbf{x}_2$ and $\lambda_2 = -1$.

The trick: make A − λI singular

Time to face the equation $A\mathbf{x} = \lambda\mathbf{x}$ in general. Here’s the trick — simple idea. Bring everything to one side:

$$(A - \lambda I)\,\mathbf{x} = \mathbf{0}$$

We don’t know $\lambda$ and we don’t know $\mathbf{x}$, but we know something: if this is to have a solution $\mathbf{x}$ that isn’t the zero vector (a zero eigenvector is useless — doesn’t count), then the matrix $A - \lambda I$ — the matrix $A$ shifted by $\lambda$ down its diagonal — must be singular. Otherwise the only solution would be $\mathbf{x} = \mathbf{0}$. And singular means determinant zero:

So the plan is: find $\lambda$ first. Then, for each eigenvalue, $A - \lambda I$ is a singular matrix and the eigenvectors are its null space — and we’re experts at finding null spaces. Elimination, identify the pivot columns, give the free variable the value one, done. Finding $\mathbf{x}$ second is a doable job.

A worked example, with trace and determinant hiding in it

Two numbers in that quadratic deserve a second look. The $6$ in $\lambda^2 - 6\lambda + 8$ is $3 + 3$ — the trace. The $8$ is the determinant of $A$, which equals the product of the eigenvalues: $4 \times 2 = 8$. In the two-by-two case it’s really nice:

$$\lambda^2 - (\text{trace})\,\lambda + (\text{determinant}) = 0$$

And since $A$ is symmetric, two promised things happened: the eigenvalues came out as nice real numbers, and the eigenvectors $(1,1)$ and $(-1,1)$ are perpendicular — dot product zero. That’s not luck; that’s the whole point, and it happened for the symmetric permutation matrix too.

Hunt for those eigenvectors yourself. Rotate $\mathbf{x}$ around the unit circle and watch $A\mathbf{x}$ swing around — it lines up with $\mathbf{x}$ only along $(1,1)$ and $(-1,1)$, where the stretch factor reads off the eigenvalue:

Adding 3I — and why A + B is not so great

Compare this example with the permutation matrix. How are they related? $\begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$ is just $\begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}$ plus $3I$. What happened to the eigenvalues? They went from $1, -1$ to $4, 2$ — three bigger. What happened to the eigenvectors? Nothing at all — still $(1,1)$ and $(-1,1)$. And it’s easy to see why: if $A\mathbf{x} = \lambda\mathbf{x}$, then

$$(A + 3I)\,\mathbf{x} = A\mathbf{x} + 3\mathbf{x} = (\lambda + 3)\,\mathbf{x}.$$

Same eigenvector $\mathbf{x}$, eigenvalue shifted to $\lambda + 3$. This is the question we’ll keep asking all chapter: do something to the matrix — what happens to its eigenvalues and eigenvectors?

But now a caution. Suppose $B$ is not $3I$ but some general matrix with eigenvalues $\alpha$. You might want to argue: $A\mathbf{x} = \lambda\mathbf{x}$ and $B\mathbf{x} = \alpha\mathbf{x}$, add them, and $A + B$ has eigenvalue $\lambda + \alpha$. That’s false. What’s the matter with the argument? We have no reason to believe $\mathbf{x}$ is also an eigenvector of $B$ — a different matrix normally has different eigenvectors, some other vector $\mathbf{y}$, “and when I add I get just rubbish.”

Rotations: when the eigenvalues go complex

Another example, chosen to bring out a possibility — something is going to go wrong. Take the matrix that rotates every vector by ninety degrees:

$$Q = \begin{bmatrix} \cos 90° & -\sin 90° \\ \sin 90° & \cos 90° \end{bmatrix} = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}$$

It deserves the letter $Q$ — a very, very orthogonal matrix. Before computing anything, the shortcuts already smell trouble. The trace is $0 + 0 = 0$, so the two eigenvalues add to zero: a plus and a minus. But the determinant is $1$, and the determinant is the product of the eigenvalues — a plus and a minus can’t multiply to $+1$. And geometrically: if every vector gets rotated by ninety degrees, what vector could possibly come out parallel to itself?

Follow the rules anyway:

$$\det(Q - \lambda I) = \begin{vmatrix} -\lambda & -1 \\ 1 & -\lambda \end{vmatrix} = \lambda^2 + 1 = 0$$

The eigenvalues are $\lambda_1 = i$ and $\lambda_2 = -i$. Complex numbers have to enter 18.06 at this moment — a perfectly real matrix can have imaginary eigenvalues. There’s a little order in the chaos: complex eigenvalues of a real matrix come in complex conjugate pairs — if $\lambda$ is an eigenvalue, so is the number with the sign of its imaginary part switched.

When does this happen? Symmetric matrices are safe: symmetric, or close to it, means real eigenvalues. But $Q$ is as far from symmetric as you can get — it’s anti-symmetric, $Q^T = -Q$: flip across the diagonal and every sign reverses. Those are the extreme case, with pure imaginary eigenvalues; general matrices sit somewhere in between.

Repeated eigenvalues: a shortage of eigenvectors

One more bad thing can happen — Strang is getting through the bad things today, “so the next lecture can be pure happiness.” Take

$$A = \begin{bmatrix} 3 & 1 \\ 0 & 3 \end{bmatrix}.$$

This matrix is triangular, and that’s really useful to know: for a triangular matrix you can read the eigenvalues right off the diagonal. Check it honestly:

$$\det(A - \lambda I) = \begin{vmatrix} 3-\lambda & 1 \\ 0 & 3-\lambda \end{vmatrix} = (3-\lambda)(3-\lambda) = 0$$

The determinant of a triangular matrix is the product down the diagonal, so $\lambda_1 = 3$ and $\lambda_2 = 3$ — a repeated eigenvalue. Why be pessimistic about a matrix whose eigenvalues can be read at a glance? The problem is in the eigenvectors. For $\lambda = 3$:

$$A - 3I = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$$

Its null space is the line through $\mathbf{x}_1 = (1, 0)$. And the eigenvector for $\lambda_2$? It’s three again — the same matrix, the same null space. We want a second, independent eigenvector, never one that depends on $\mathbf{x}_1$, and the conclusion is: there isn’t one. This is a degenerate matrix — a two-by-two matrix with only one line of eigenvectors instead of two. A repeated eigenvalue opens the possibility of a shortage of eigenvectors, and matrices like this are the ones where eigenvectors won’t give the complete story. Monday’s lecture gives the complete story for all the others.

Problems

Work these before revealing the solutions — each uses only this lecture’s methods.

Problem 21.1 Full eigenvalue computation

Find the eigenvalues and eigenvectors of

$$A = \begin{bmatrix} 5 & 1 \\ 1 & 5 \end{bmatrix}.$$

Check your eigenvalues against the trace and the determinant, and check that the eigenvectors are perpendicular.

Show solution

$$\det(A - \lambda I) = (5-\lambda)^2 - 1 = \lambda^2 - 10\lambda + 24 = (\lambda - 6)(\lambda - 4) = 0$$

so $\lambda_1 = 6$, $\lambda_2 = 4$. Checks: trace $= 5 + 5 = 10 = 6 + 4$ ✓ and $\det A = 25 - 1 = 24 = 6 \times 4$ ✓.

For $\lambda_1 = 6$: $A - 6I = \begin{bmatrix} -1 & 1 \\ 1 & -1 \end{bmatrix}$, whose null space gives $\mathbf{x}_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$.

For $\lambda_2 = 4$: $A - 4I = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$, whose null space gives $\mathbf{x}_2 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}$.

Perpendicular: $(1)(-1) + (1)(1) = 0$ ✓ — as they must be, since $A$ is symmetric.

Problem 21.2 Eigenvalues from the picture

The matrix

$$P = \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$

projects every vector in the plane onto the line through $(1, 1)$. Without writing the characteristic equation, use the projection picture to find both eigenvalues and their eigenvectors. Then verify the trace.

Show solution

A vector already on the line is unchanged by the projection: $P\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$, so $\mathbf{x}_1 = (1,1)$ has eigenvalue $\lambda_1 = 1$.

A vector perpendicular to the line projects to zero: $P\begin{bmatrix} -1 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$, so $\mathbf{x}_2 = (-1, 1)$ has eigenvalue $\lambda_2 = 0$ — it’s in the null space, and $P$ is singular.

Trace check: $\frac{1}{2} + \frac{1}{2} = 1 = 1 + 0$ ✓.

Problem 21.3 Trace, determinant, and shifting

A $2 \times 2$ matrix $A$ has trace $7$ and determinant $12$.

(a) Find the eigenvalues of $A$ without knowing its entries.

(b) What are the eigenvalues of $A + 2I$? What happens to the eigenvectors?

Show solution

(a) The eigenvalues satisfy $\lambda^2 - (\text{trace})\lambda + \det A = \lambda^2 - 7\lambda + 12 = (\lambda - 3)(\lambda - 4) = 0$, so $\lambda_1 = 3$ and $\lambda_2 = 4$. (Sum $3 + 4 = 7$, product $3 \times 4 = 12$ ✓.)

(b) If $A\mathbf{x} = \lambda\mathbf{x}$, then $(A + 2I)\mathbf{x} = A\mathbf{x} + 2\mathbf{x} = (\lambda + 2)\mathbf{x}$. So the eigenvalues shift to $5$ and $6$, and the eigenvectors are exactly the same vectors $\mathbf{x}$.

Problem 21.4 Complex eigenvalues

Find the eigenvalues of

$$A = \begin{bmatrix} 0 & -2 \\ 2 & 0 \end{bmatrix}.$$

Before computing, predict from the trace and determinant why they can’t be real, and say what property of $A$ guarantees pure imaginary eigenvalues.

Show solution

Prediction: trace $= 0$, so $\lambda_1 + \lambda_2 = 0$ — one plus, one minus. But $\det A = 0 - (-4) = 4 = \lambda_1\lambda_2$, and two real numbers that add to zero have a product $\leq 0$. No real pair works.

Compute: $\det(A - \lambda I) = \lambda^2 + 4 = 0$, so $\lambda = 2i$ and $\lambda = -2i$ — a complex conjugate pair, as complex eigenvalues of a real matrix must be.

The guarantee: $A$ is anti-symmetric, $A^T = -A$, the extreme opposite of symmetric — those are exactly the matrices with pure imaginary eigenvalues.

Problem 21.5 Shortage of eigenvectors

Find the eigenvalues and all independent eigenvectors of the triangular matrix

$$A = \begin{bmatrix} 4 & 5 \\ 0 & 4 \end{bmatrix}.$$

Is there a full set of two independent eigenvectors?

Show solution

Triangular matrix: the eigenvalues sit on the diagonal, $\lambda_1 = \lambda_2 = 4$ — repeated. Honestly: $\det(A - \lambda I) = (4 - \lambda)^2 = 0$.

For $\lambda = 4$:

$$A - 4I = \begin{bmatrix} 0 & 5 \\ 0 & 0 \end{bmatrix}$$

The null space forces $5x_2 = 0$, so $x_2 = 0$ and the free variable is $x_1$: the only eigenvectors are multiples of $\mathbf{x}_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$.

Both eigenvalues give the same singular matrix and the same null space, so there is no second independent eigenvector. The matrix is degenerate: one line of eigenvectors instead of two.