Dimensionality reduction: PCA, t-SNE and UMAP

Why reduce dimensions

Three reasons: compression and speed (fewer features for downstream kNN or clustering), denoising (small-variance directions are often noise), and visualization (we can only see 2–3 dimensions). The methods split into linear projections that preserve global geometry (PCA) and nonlinear embeddings that preserve neighbourhoods (t-SNE, UMAP).

PCA as variance maximization

Center the data so each feature has mean zero and stack it as $\mathbf{X}\in\mathbb{R}^{n\times d}$. The covariance matrix is $\mathbf{S}=\frac1n\mathbf{X}^\top\mathbf{X}$ (symmetric, positive semi-definite). We want the unit direction $\mathbf{u}$ along which the projected data $\mathbf{u}^\top\mathbf{x}_i$ has the largest variance:

The Lagrangian $\mathbf{u}^\top\mathbf{S}\mathbf{u}-\lambda(\mathbf{u}^\top\mathbf{u}-1)$ has gradient $2\mathbf{S}\mathbf{u}-2\lambda\mathbf{u}=0$, so $\mathbf{S}\mathbf{u}=\lambda\mathbf{u}$: the optimal direction is an eigenvector of the covariance, and the variance it captures is $\mathbf{u}^\top\mathbf{S}\mathbf{u}=\lambda$. The first principal component is the eigenvector with the largest eigenvalue; the second is the top eigenvector orthogonal to the first, and so on. The explained variance ratio of component $k$ is $\lambda_k/\sum_j\lambda_j$.

Worked example. Two features with covariance $\mathbf{S}=\begin{bmatrix}2&1\\1&2\end{bmatrix}$. Eigenvalues solve $(2-\lambda)^2-1=0$, giving $\lambda_1=3$, $\lambda_2=1$, with eigenvectors $\frac{1}{\sqrt2}(1,1)$ and $\frac{1}{\sqrt2}(1,-1)$. The data is elongated along the diagonal; PC1 explains $3/4=75\%$ of the variance, and projecting onto it keeps most of the structure with one number per point.

PCA as reconstruction, and via SVD

The same components solve a different problem: find the $k$-dimensional subspace whose orthogonal projection $\mathbf{U}_k\mathbf{U}_k^\top\mathbf{x}_i$ best reconstructs the data, minimizing $\sum_i\|\mathbf{x}_i-\mathbf{U}_k\mathbf{U}_k^\top\mathbf{x}_i\|^2$. This is the Eckart–Young theorem from 1.3 Eigen-things, SVD and low rank: the best rank-$k$ approximation of $\mathbf{X}$ comes from its SVD, $\mathbf{X}=\mathbf{U}\boldsymbol\Sigma\mathbf{V}^\top$. The principal directions are the columns of $\mathbf{V}$, the projected coordinates ("scores") are $\mathbf{X}\mathbf{V}_k=\mathbf{U}_k\boldsymbol\Sigma_k$, and the eigenvalues are $\lambda_j=\sigma_j^2/n$. Libraries compute PCA through the SVD because forming $\mathbf{X}^\top\mathbf{X}$ squares the condition number. Whitening additionally rescales each component to unit variance, $\mathbf{z}=\boldsymbol\Lambda^{-1/2}\mathbf{V}^\top\mathbf{x}$, which decorrelates and equalizes features.

Intuition. PCA rotates your coordinate system to align with the data's own axes of spread, then drops the axes with the least spread. Nothing about the labels is involved — PCA is unsupervised.

Using PCA well

Center always; standardize when features have different units, otherwise the largest-unit feature becomes PC1. Choose $k$ by cumulative explained variance (90–95%) or a scree-plot elbow. PCA before kNN or k-means often helps (less noise, faster distances); PCA as preprocessing for a supervised model can hurt, because variance is not the same as predictive relevance — a low-variance direction may carry the label. PCA is linear and sensitive to outliers; kernel PCA and robust PCA address those. Fit PCA on the training fold only inside a pipeline (3.13 Feature engineering, data quality and data-centric ML).

t-SNE: preserving neighbourhoods

t-SNE builds a map in 2-D where nearby points stay nearby. In the input space it converts distances to conditional probabilities,

where each $\sigma_i$ is set so the distribution has a user-chosen perplexity (effective number of neighbours, typically 5–50). In the map it uses a heavy-tailed Student-$t$ kernel, $q_{ij}\propto(1+\|\mathbf{y}_i-\mathbf{y}_j\|^2)^{-1}$, and moves the points $\mathbf{y}_i$ by gradient descent to minimize $\mathrm{KL}(P\,\|\,Q)$ (1.9 Information theory: entropy, cross-entropy and KL divergence). The heavy tails fix the crowding problem: moderately distant points can be pushed far apart in 2-D without a large penalty. Consequences you must respect (Wattenberg et al., 2016): cluster sizes mean nothing, distances between clusters mean little, random noise can look clustered at low perplexity, and the result depends on the seed. Run several perplexities, and never cluster or measure on a t-SNE plot. t-SNE has no transform for new points.

UMAP

UMAP builds a weighted $k$-nearest-neighbour graph with locally adaptive distances (so every point is connected to its neighbours), then optimizes a low-dimensional layout whose own neighbour graph matches it, minimizing a cross-entropy between edge weights. It is much faster than t-SNE, scales to millions of points, keeps somewhat more global structure, and provides transform for new data. Its knobs are n_neighbors (local vs global focus) and min_dist (how tightly points pack). The same caveats about reading distances apply.

Autoencoders and low rank

A linear autoencoder with $k$ hidden units trained with squared error learns the same subspace as PCA with $k$ components (4.10 Autoencoders, embeddings and representation learning); add nonlinearities and you get nonlinear PCA. Low-rank structure is everywhere later: LoRA fine-tunes an LLM by learning a rank-$r$ update (8.2 Fine-tuning and PEFT), attention heads are low-rank bilinear maps (7.2 Scaled dot-product and multi-head attention, fully derived), and Matryoshka embeddings are trained so their first $k$ coordinates behave like a PCA truncation (9.2 Embeddings deep dive).

Where this shows up later

• Eigenvectors, SVD and Eckart–Young — 1.3 Eigen-things, SVD and low rank; clustering in reduced spaces — 3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical; pipeline hygiene — 3.13 Feature engineering, data quality and data-centric ML.
• Autoencoders as nonlinear PCA — 4.10 Autoencoders, embeddings and representation learning; visualizing word and sentence embeddings — 6.3 Word embeddings, 9.2 Embeddings deep dive; PCA/probing on model activations in interpretability — 8.11 Interpretability: what is happening inside; low-rank adaptation — 8.2 Fine-tuning and PEFT.