Eigen-things, SVD and low rank

Directions a matrix does not rotate

Most vectors change direction under $\mathbf{A}$. An eigenvector $\mathbf{v}\neq\mathbf{0}$ does not: $\mathbf{A}\mathbf{v}=\lambda\mathbf{v}$, it is only scaled by the eigenvalue $\lambda$. Rearranging, $(\mathbf{A}-\lambda\mathbf{I})\mathbf{v}=\mathbf{0}$ has a nonzero solution only if $\mathbf{A}-\lambda\mathbf{I}$ squashes space, that is, $\det(\mathbf{A}-\lambda\mathbf{I})=0$ — the characteristic equation. For

For $\lambda=3$, $(\mathbf{A}-3\mathbf{I})\mathbf{v}=\begin{bmatrix}-1&1\\1&-1\end{bmatrix}\mathbf{v}=\mathbf{0}$ gives $\mathbf{v}_1\propto(1,1)$; for $\lambda=1$, $\mathbf{v}_2\propto(1,-1)$. Along the diagonal the matrix stretches by 3, along the anti-diagonal it leaves lengths alone, and every other vector is a mixture of the two behaviors. Set these numbers in the demo and watch the eigenvector arrows stay on their lines while the grid shears around them.

Intuition. Eigenvectors are the axes along which a transformation is "just scaling". Describe the matrix in that coordinate system and it becomes diagonal — the simplest kind of matrix.

Diagonalization and repeated application

If an $n\times n$ matrix has $n$ independent eigenvectors, stack them as columns of $\mathbf{V}$ and put the eigenvalues on the diagonal of $\mathbf{\Lambda}$; then $\mathbf{A}\mathbf{V}=\mathbf{V}\mathbf{\Lambda}$, so

Read it as: change to the eigenbasis, scale axis $i$ by $\lambda_i^k$, change back. Powers of a matrix are therefore governed by the largest $|\lambda|$: components along that eigenvector grow like $\lambda_{\max}^k$ and dominate everything else — the idea behind power iteration, Google's original PageRank and the long-run behavior of Markov chains — while components with $|\lambda|<1$ die out geometrically. This is the mathematical heart of vanishing and exploding gradients: backpropagation through $T$ steps of a recurrent network multiplies $T$ nearly identical Jacobians, and the result is either $\sim\lambda_{\max}^T$ (explodes) or $\sim\lambda_{\max}^T\to0$ (vanishes) (6.4 Recurrent neural networks and BPTT, 4.6 Initialization and normalization). Two free checks on any eigen-computation: $\sum_i\lambda_i=\operatorname{tr}\mathbf{A}=\sum_iA_{ii}$ and $\prod_i\lambda_i=\det\mathbf{A}$. Above: $3+1=4=2+2$ ✓ and $3\cdot1=3=4-1$ ✓.

Symmetric matrices and the spectral theorem

When $\mathbf{A}=\mathbf{A}^\top$, everything is nicer. The spectral theorem says all eigenvalues are real and the eigenvectors can be chosen orthonormal, so

The quadratic form is a weighted sum of squared coordinates in the eigenbasis, so $\mathbf{A}$ is positive semidefinite exactly when every $\lambda_i\ge0$ and positive definite when all are $>0$. Covariance matrices — $\mathbf{\Sigma}=\tfrac1n\mathbf{X}^\top\mathbf{X}$ for centered data — are symmetric PSD, and their eigenvectors are the principal axes of the data cloud with eigenvalues equal to the variances along them (3.12 Dimensionality reduction: PCA, t-SNE and UMAP, 1.7 Probability II: Bayes, independence, MLE and MAP). Hessians are symmetric, and their eigenvalues are the curvatures along the eigen-directions: all positive at a minimum, all negative at a maximum, mixed at a saddle (1.5 Multivariable and matrix calculus, 1.10 Optimization: from gradient descent to Adam). Kernel matrices in SVMs are PSD by construction (3.10 Support vector machines and kernels).

The SVD: every matrix is rotate, stretch, rotate

Eigen-decomposition needs a square (and diagonalizable) matrix. The singular value decomposition works for any $\mathbf{A}\in\mathbb{R}^{m\times n}$:

with $\mathbf{U}$ ($m\times m$) and $\mathbf{V}$ ($n\times n$) orthogonal and $\mathbf{\Sigma}$ diagonal holding the singular values $\sigma_1\ge\sigma_2\ge\dots\ge0$. Geometrically: rotate by $\mathbf{V}^\top$, stretch axis $i$ by $\sigma_i$, rotate by $\mathbf{U}$ — so the unit sphere always maps to an ellipsoid whose semi-axes are the $\sigma_i$. The number of nonzero singular values is the rank $r$, and the sum form shows $\mathbf{A}$ as a sum of $r$ rank-one matrices. The SVD is tied to eigen-decomposition through

so the right singular vectors are the eigenvectors of $\mathbf{A}^\top\mathbf{A}$ and $\sigma_i^2$ are its eigenvalues; for a symmetric PSD matrix the singular values are the eigenvalues. The ratio $\kappa=\sigma_{\max}/\sigma_{\min}$ is the condition number: how unevenly the matrix stretches, and how much a linear solve can amplify rounding error. Forming $\mathbf{X}^\top\mathbf{X}$ squares the condition number, which is why least-squares solvers use QR or the SVD rather than the normal equations (3.2 Linear regression).

Low rank: Eckart–Young and why compression works

Keep only the $k$ largest terms: $\mathbf{A}_k=\sum_{i\le k}\sigma_i\mathbf{u}_i\mathbf{v}_i^\top$. The Eckart–Young theorem says $\mathbf{A}_k$ is the best rank-$k$ approximation of $\mathbf{A}$ in both the Frobenius and the spectral norm, with error

Storage drops from $mn$ numbers to $k(m+n)$: a $1000\times1000$ image kept at rank 50 needs 100,000 numbers instead of a million, and if the singular values decay quickly you barely see the difference. The same theorem explains PCA (project onto the top singular directions of centered data, 3.12 Dimensionality reduction: PCA, t-SNE and UMAP), why embedding tables and weight matrices are compressible, and why LoRA can adapt a huge model by learning only a rank-$r$ update $\Delta\mathbf{W}=\mathbf{B}\mathbf{A}$ with $r(d_{\text{in}}+d_{\text{out}})$ parameters instead of $d_{\text{in}}d_{\text{out}}$ (8.2 Fine-tuning and PEFT).

Worked example: the SVD of a shear

Let $\mathbf{A}=\begin{bmatrix}1&1\\0&1\end{bmatrix}$. Then $\mathbf{A}^\top\mathbf{A}=\begin{bmatrix}1&1\\1&2\end{bmatrix}$ with characteristic equation $\lambda^2-3\lambda+1=0$, so

the golden ratio and its reciprocal. Checks: $\sigma_1\sigma_2=1=\det\mathbf{A}$ ✓ and $\sigma_1^2+\sigma_2^2=3=\|\mathbf{A}\|_F^2$ (the sum of squared entries) ✓. The condition number is $\kappa\approx2.618$: a shear that looks mild still stretches one direction 2.6 times more than another. The best rank-one approximation $\sigma_1\mathbf{u}_1\mathbf{v}_1^\top$ has relative error $\sigma_2/\|\mathbf{A}\|_F\approx0.618/1.732\approx0.36$.

Math check. The right singular vector for $\sigma_1$ solves $(\mathbf{A}^\top\mathbf{A}-2.618\,\mathbf{I})\mathbf{v}=\mathbf{0}$, giving $\mathbf{v}_1\propto(1,\,1.618)$; then $\mathbf{u}_1=\mathbf{A}\mathbf{v}_1/\sigma_1\propto(2.618,\,1.618)/1.618\propto(1.618,\,1)$. Unit-normalize both and you have the first term of the SVD.

import numpy as np
A = np.array([[1., 1.], [0., 1.]])
U, s, Vt = np.linalg.svd(A)
print(s)                     # [1.618 0.618]
A1 = s[0] * np.outer(U[:, 0], Vt[0])          # best rank-1 approximation
print(np.linalg.norm(A - A1) / np.linalg.norm(A))   # 0.357

Where this shows up later

• PCA is the SVD of centered data — 3.12 Dimensionality reduction: PCA, t-SNE and UMAP; spectral ideas in clustering — 3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical.
• Conditioning governs gradient descent's speed — 1.10 Optimization: from gradient descent to Adam; Hessian eigenvalues are curvatures — 1.5 Multivariable and matrix calculus; repeated Jacobians vanish or explode — 6.4 Recurrent neural networks and BPTT, 4.6 Initialization and normalization.
• Low-rank adapters — 8.2 Fine-tuning and PEFT; attention score matrices have rank at most $d_k$ — 7.2 Scaled dot-product and multi-head attention, fully derived; covariance ellipses of the Gaussian — 1.7 Probability II: Bayes, independence, MLE and MAP.