Symmetric Matrices and Positive Definiteness

The two main facts, stated right away

This is the lecture on symmetric matrices — “the most important class of matrices.” The question we now ask of any matrix is: what are its eigenvalues, what are its eigenvectors? And when a matrix has special structure, that structure should show up there. Markov matrices have an eigenvalue equal to $1$. What do symmetric matrices — real matrices with $A = A^T$ — have?

Strang tells you the two main facts immediately:

1. The eigenvalues are real. The rotation matrices that produced complex eigenvalues — that can’t happen here.
2. The eigenvectors are perpendicular — or more carefully, can be chosen perpendicular.

Why the careful wording? Take the identity matrix: certainly symmetric, all eigenvalues equal to $1$. But every vector is an eigenvector, so how can they all be perpendicular? The point is that in the typical case — all eigenvalues different — each eigenvalue owns one line of eigenvectors, and those lines really are perpendicular. When an eigenvalue repeats, there’s a whole plane of eigenvectors, and in that plane we have the freedom to choose perpendicular ones. Either way, a symmetric matrix has a full set of perpendicular eigenvectors.

The spectral theorem

For a usual matrix with a complete set of eigenvectors, diagonalization reads $A = S \Lambda S^{-1}$. What changes when $A$ is symmetric? The $\Lambda$ is still diagonal, with real $\lambda$‘s. But the eigenvector matrix is now special: its columns are perpendicular, and we can scale each to length one — no problem. So the columns are orthonormal, and the right letter is no longer $S$ but $Q$:

$$A = Q \Lambda Q^{-1} = Q \Lambda Q^T$$

The last step is the payoff: for a square matrix with orthonormal columns, the inverse is the transpose, $Q^{-1} = Q^T$.

The name comes from the spectrum — the set of eigenvalues — by analogy with the spectrum of light as a combination of pure things: the matrix is broken into pure eigenvalues and eigenvectors. In mechanics it’s the principal axis theorem: look at a material along the right axes and it becomes diagonal — the directions don’t couple.

And the symmetry is visible in the formula itself. Transpose $Q \Lambda Q^T$: the outer factors swap, $\Lambda^T = \Lambda$, and you get the same product back. Anything of that form is automatically symmetric, and the theorem says every symmetric matrix arises this way.

Why the eigenvalues are real

Start from the only thing we know: $A\mathbf{x} = \lambda \mathbf{x}$, where for all we know at this moment $\lambda$ and $\mathbf{x}$ could be complex. Two moves, then a comparison.

Move 1 — conjugate everything. Conjugation respects products, so $\bar{A}\bar{\mathbf{x}} = \bar{\lambda}\bar{\mathbf{x}}$. Since $A$ is real, $\bar{A} = A$:

$$A\bar{\mathbf{x}} = \bar{\lambda}\bar{\mathbf{x}}$$

Notice symmetry hasn’t been used yet — this already says something on its own: for any real matrix, complex eigenvalues come in conjugate pairs $\lambda, \bar{\lambda}$, with eigenvectors $\mathbf{x}, \bar{\mathbf{x}}$.

Move 2 — transpose. Transposing the conjugated equation gives $\bar{\mathbf{x}}^T A^T = \bar{\mathbf{x}}^T \bar{\lambda}$. Now symmetry enters: $A^T = A$. “At that moment I used the assumption.”

Compare. Multiply the original equation on the left by $\bar{\mathbf{x}}^T$, and multiply the transposed equation on the right by $\mathbf{x}$:

$$\bar{\mathbf{x}}^T A \mathbf{x} = \lambda\, \bar{\mathbf{x}}^T \mathbf{x} \qquad \text{and} \qquad \bar{\mathbf{x}}^T A \mathbf{x} = \bar{\lambda}\, \bar{\mathbf{x}}^T \mathbf{x}$$

Same left side, so the right sides agree: $\lambda\, \bar{\mathbf{x}}^T\mathbf{x} = \bar{\lambda}\, \bar{\mathbf{x}}^T\mathbf{x}$. If the factor $\bar{\mathbf{x}}^T\mathbf{x}$ is not zero, cancel it and conclude $\lambda = \bar{\lambda}$ — a number equal to its own conjugate has no imaginary part. $\lambda$ is real.

So everything hangs on that one quantity, and it’s worth a second of its own. For a real vector, $\mathbf{x}^T\mathbf{x} = x_1^2 + x_2^2 + \cdots$ is the length squared — Pythagoras. For a complex vector the honest analogue conjugates one factor: each term is $\bar{x}_k x_k$, and when you multiply $a + ib$ by $a - ib$, the result is $a^2 + b^2$ — the $iab$ and $-iab$ cancel, and $i \cdot (-i)$ makes the $b^2$ come in with a plus. Every term is positive, so $\bar{\mathbf{x}}^T\mathbf{x}$ is positive except for the zero vector. “If you want a decent answer, multiply numbers by their conjugates.” That’s the length squared of a complex vector, and it’s the safe cancellation that finishes the proof — and a preview of the next lecture, where complex vectors and matrices get their full treatment.

Every symmetric matrix is a combination of projections

Break the factorization down. Write out $Q \Lambda Q^T$ with columns $\mathbf{q}_1, \mathbf{q}_2, \ldots$ and multiply columns times rows:

$$A = Q\Lambda Q^T = \lambda_1 \mathbf{q}_1 \mathbf{q}_1^T + \lambda_2 \mathbf{q}_2 \mathbf{q}_2^T + \cdots + \lambda_n \mathbf{q}_n \mathbf{q}_n^T$$

What kind of matrix is $\mathbf{q}\mathbf{q}^T$, a unit column times its own transpose? We met it in chapter four: it’s $\mathbf{a}\mathbf{a}^T$ with no need to divide by $\mathbf{a}^T\mathbf{a}$, since that’s $1$. It’s a projection matrix — symmetric, and squaring it brings back a $\mathbf{q}^T\mathbf{q} = 1$ in the middle, so it equals its own square.

So every symmetric matrix is a combination of mutually perpendicular projection matrices, weighted by real eigenvalues. That’s another way people like to state the spectral theorem.

Pivots know the signs of the eigenvalues

Once the eigenvalues are real, the next question is: are they positive or negative? That’s the question that decides between stability and instability in differential equations. And you’d hate to have to compute the eigenvalues to answer it. For a symmetric matrix of order 50, finding the 50 eigenvalues by pencil and paper is a lifetime’s job — and even numerically, forming the degree-50 polynomial $\det(A - \lambda I)$ and finding its roots is a very bad, unstable way to do it. (Strang apologizes: that’s the method he taught, but he meant it for two-by-twos and three-by-threes.) Real eigenvalue computation belongs to numerical linear algebra.

But Matlab will quite happily find the 50 pivots, safely and quickly. And here’s the remarkable fact:

Nobody is confusing pivots with eigenvalues — they’re different numbers. The one connection you already know is that the product of the pivots equals the product of the eigenvalues, because both equal the determinant (pivots when there are no row exchanges). But the product says nothing about the 50 individual signs, which this theorem does.

It even gives a start on computing eigenvalues: shift the matrix to $A - 7I$, and all eigenvalues shift down by $7$. Count the positive pivots of the shifted matrix and you know how many eigenvalues of $A$ lie above $7$ and how many below.

Positive definite matrices: the excellent subclass

The last minutes start the next section of the book. If symmetric matrices are good, positive definite matrices are “a subclass that are excellent — just the greatest.”

What about determinants as a test? Positive eigenvalues certainly make the determinant positive — but that alone is not enough. The matrix

$$\begin{bmatrix} -1 & 0 \\ 0 & -3 \end{bmatrix}$$

has determinant $+3$, yet its pivots (and eigenvalues) are $-1$ and $-3$: not positive definite at all. The fix is to test all the sub-determinants coming down from the upper left — $n$ tests, not one. For the good example: $5 > 0$ and $11 > 0$, passes. For the bad one: the $1 \times 1$ determinant is $-1$, fails immediately.

Problems

Work these before revealing the solutions — the pivot tests make most of them fast.

Problem 25.1 Spectral factorization by hand

Find the eigenvalues and orthonormal eigenvectors of

$$A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$$

and write out the factorization $A = Q\Lambda Q^T$. Check that the eigenvectors are perpendicular.

Show solution

$\det(A - \lambda I) = \lambda^2 - 6\lambda + 8 = (\lambda - 4)(\lambda - 2)$, so $\lambda_1 = 4$, $\lambda_2 = 2$ — real, as symmetry guarantees.

For $\lambda = 4$: $(A - 4I)\mathbf{x} = \mathbf{0}$ gives $\mathbf{x}_1 = (1, 1)$. For $\lambda = 2$: $\mathbf{x}_2 = (1, -1)$. Their dot product is $1 - 1 = 0$: perpendicular. Normalize to unit length (divide by $\sqrt{2}$):

$$Q = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}, \qquad \Lambda = \begin{bmatrix} 4 & 0 \\ 0 & 2 \end{bmatrix}, \qquad A = Q \Lambda Q^T.$$

Multiplying it out: $Q\Lambda Q^T = \frac{1}{2}\begin{bmatrix} 4 + 2 & 4 - 2 \\ 4 - 2 & 4 + 2 \end{bmatrix} = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$. ✓

Problem 25.2 Projection-matrix form

For the matrix $A$ of Problem 25.1, write $A$ as a combination $\lambda_1 \mathbf{q}_1\mathbf{q}_1^T + \lambda_2 \mathbf{q}_2\mathbf{q}_2^T$ of projection matrices, and verify that each $\mathbf{q}\mathbf{q}^T$ equals its own square.

Show solution

With $\mathbf{q}_1 = \frac{1}{\sqrt{2}}(1,1)$ and $\mathbf{q}_2 = \frac{1}{\sqrt{2}}(1,-1)$:

$$\mathbf{q}_1\mathbf{q}_1^T = \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}, \qquad \mathbf{q}_2\mathbf{q}_2^T = \frac{1}{2}\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix}$$

Then

$$4 \cdot \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} + 2 \cdot \frac{1}{2}\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 2 \\ 2 & 2 \end{bmatrix} + \begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} = A. ✓$$

Each piece is a projection: squaring $\mathbf{q}\mathbf{q}^T$ gives $\mathbf{q}(\mathbf{q}^T\mathbf{q})\mathbf{q}^T = \mathbf{q}\mathbf{q}^T$ because $\mathbf{q}^T\mathbf{q} = 1$. (Check on the first one: $\frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$ squared is $\frac{1}{4}\begin{bmatrix} 2 & 2 \\ 2 & 2 \end{bmatrix}$ — the same matrix.)

Problem 25.3 Counting eigenvalue signs by pivots

Without computing any eigenvalues, determine how many eigenvalues of

$$A = \begin{bmatrix} 1 & 3 \\ 3 & 1 \end{bmatrix}$$

are positive and how many are negative. Then confirm by finding the eigenvalues.

Show solution

Eliminate: the first pivot is $1$. Subtract $3$ times row one from row two: the second pivot is $1 - 9 = -8$. One positive pivot, one negative pivot — so by the sign theorem for symmetric matrices, one positive and one negative eigenvalue.

Confirm: $\lambda^2 - 2\lambda + (1 - 9) = \lambda^2 - 2\lambda - 8 = (\lambda - 4)(\lambda + 2)$, so $\lambda = 4$ and $\lambda = -2$. One positive, one negative, matching the pivots. (Consistency check: product of pivots $= 1 \cdot (-8) = -8 = \det A =$ product of eigenvalues $= 4 \cdot (-2)$.)

Problem 25.4 Testing positive definiteness

For which numbers $b$ is

$$A = \begin{bmatrix} 4 & b \\ b & 9 \end{bmatrix}$$

positive definite? Answer with the sub-determinant test, and check that the pivot test gives the same condition.

Show solution

Sub-determinants: the $1\times 1$ determinant is $4 > 0$ ✓; the $2\times 2$ determinant is $36 - b^2$, which is positive exactly when $-6 < b < 6$.

Pivots: the first pivot is $4 > 0$. The second pivot is the determinant divided by the first pivot, $\frac{36 - b^2}{4}$, positive under exactly the same condition.

So $A$ is positive definite precisely for $-6 < b < 6$. At $b = \pm 6$ the determinant is zero — the matrix is singular, with an eigenvalue $0$, so it just fails.

Problem 25.5 Determinant alone is not enough

Give a symmetric $2\times 2$ matrix whose determinant is positive but which is not positive definite, and explain which test it fails. Then state what its two pivots tell you about its two eigenvalues.

Show solution

The lecture’s own example works:

$$A = \begin{bmatrix} -1 & 0 \\ 0 & -3 \end{bmatrix}, \qquad \det A = 3 > 0.$$

It fails the sub-determinant test at the first step: the $1\times 1$ upper-left determinant is $-1$, which is negative. (Two negatives multiply to a positive determinant — that’s exactly why the whole determinant alone can’t detect the problem.)

The pivots are $-1$ and $-3$, both negative — so by the sign theorem both eigenvalues are negative (here they are simply $-1$ and $-3$). This matrix is in fact negative definite: the mirror image of excellent.