Autoencoders, embeddings and representation learning

Encoder, bottleneck, decoder

An autoencoder learns to copy its input through a constriction. An encoder $f_\phi$ maps the input $\mathbf{x}\in\mathbb{R}^d$ to a latent code $\mathbf{z}=f_\phi(\mathbf{x})\in\mathbb{R}^k$, and a decoder $g_\theta$ maps the code back to a reconstruction $\hat{\mathbf{x}}=g_\theta(\mathbf{z})$. Training minimizes the reconstruction error,

or a binary cross-entropy per pixel when inputs are in $[0,1]$ (4.3 Loss functions catalog). If $k\ge d$ the model could learn the identity and nothing interesting would happen; the undercomplete case $k<d$ forces the code to keep only what matters for reconstruction. Everything the network cannot afford to store — noise, redundancy, pixel-level detail predictable from context — is discarded. The code is therefore a representation: a compressed description in coordinates the model chose itself. No labels were needed, which makes autoencoders the prototype of self-supervised learning (5.8 Self-supervised and contrastive learning).

Linear autoencoders are PCA

Make both maps linear, $\mathbf{z}=\mathbf{W}_e\mathbf{x}$, $\hat{\mathbf{x}}=\mathbf{W}_d\mathbf{z}$, with squared loss. The optimal reconstruction $\mathbf{W}_d\mathbf{W}_e$ is the projection onto the subspace spanned by the top-$k$ principal components of the data (3.12 Dimensionality reduction: PCA, t-SNE and UMAP) — the Eckart–Young theorem of 1.3 Eigen-things, SVD and low rank in disguise. (Gradient descent recovers the subspace, not necessarily the individual ordered components, because any invertible change of basis inside the subspace gives the same loss.) So an autoencoder with a nonlinear encoder and decoder is nonlinear PCA: it can bend the subspace into a curved surface, which is what real data needs.

Worked example. An MNIST autoencoder $784\to32\to784$ has $(784+1)\cdot32 + (32+1)\cdot784 = 25{,}120+25{,}872 = 50{,}992$ parameters and compresses each image $24.5\times$. With linear layers its reconstruction error matches PCA with 32 components; with a ReLU hidden layer of 128 units on each side it reconstructs noticeably sharper digits at the same code size — the manifold of handwritten digits is curved.

The manifold hypothesis

Random $28\times28$ pixel arrays never look like digits; real digits occupy a thin, curved sheet inside $\mathbb{R}^{784}$ whose intrinsic dimension is perhaps a few dozen (stroke thickness, slant, loop size…). The manifold hypothesis says natural data concentrates near such low-dimensional manifolds, and the latent space is a coordinate chart for it: moving along $\mathbf{z}$ moves along the manifold, so interpolating between two codes produces plausible intermediates rather than a pixel blend. This picture explains why generative models work at all (8.8 Generative models I: VAEs and GANs, 8.9 Generative models II: Diffusion and flow matching) and why adversarial examples exist (small steps off the manifold, 5.9 Interpretability, robustness and adversarial examples).

Variants that learn better representations

• Denoising autoencoders corrupt the input, $\tilde{\mathbf{x}}=\mathbf{x}+\boldsymbol{\epsilon}$ or with random masking, and train $g(f(\tilde{\mathbf{x}}))\approx\mathbf{x}$. The model must learn the structure of the data to undo corruption, so the code cannot be a trivial copy even without a bottleneck. The optimal denoiser points from the corrupted input back toward the data manifold — it estimates the score $\nabla_{\mathbf{x}}\log p(\mathbf{x})$, which is exactly what a diffusion model learns at every noise level (8.9 Generative models II: Diffusion and flow matching).
• Sparse autoencoders add an $\ell_1$ penalty on the code (or a KL penalty toward a target activation rate) so that only a few latent units fire for any input. Over-complete sparse codes ($k\gg d$) are the tool interpretability researchers use to pull human-readable features out of LLM activations (8.11 Interpretability: what is happening inside).
• Masked autoencoders hide a large fraction of input patches or tokens and reconstruct them: MAE masks 75% of image patches (5.8 Self-supervised and contrastive learning), and BERT's masked language modeling is a denoising autoencoder over text (7.5 Encoder-only: BERT and friends (and sentence embeddings)).
• Variational autoencoders replace the point code with a distribution $q_\phi(\mathbf{z}\mid\mathbf{x})$ and add a KL term pulling it toward a prior, which turns the autoencoder into a generative model with a smooth, sample-able latent space (8.8 Generative models I: VAEs and GANs).
• Contractive autoencoders penalize the encoder's Jacobian so small input changes give small code changes — robustness by construction.

Embeddings: a learned lookup table

Discrete objects — words, tokens, users, products — have no natural coordinates. An embedding layer gives them some: nn.Embedding(V, d) is a $V\times d$ matrix $\mathbf{E}$, and the embedding of item $i$ is its row $\mathbf{E}_i$. Equivalently, it is a linear layer applied to a one-hot vector, $\mathbf{E}^\top\mathbf{e}_i=\mathbf{E}_i$, implemented as a lookup because multiplying by one-hots would be wasteful (1.2 Matrices as transformations). The rows are ordinary parameters trained by backprop through whatever task sits on top: next-token prediction (6.3 Word embeddings, 7.6 Decoder-only: GPT and autoregressive generation), click prediction, or a contrastive objective (4.3 Loss functions catalog).

Why does geometry in this space mean anything? Because every objective that uses embeddings scores them with dot products (1.1 Vectors, dot products and norms): a word's embedding is pushed toward the contexts it appears in (word2vec), a query embedding toward its relevant documents (retrieval), an image embedding toward its caption (CLIP). Items that are used similarly are pushed to similar places, so cosine similarity between embeddings measures functional similarity under the training task — not some innate semantics. Change the task and the geometry changes; that is why a retrieval model's embeddings outperform a language model's hidden states for search (9.2 Embeddings deep dive). Embeddings are the universal interface of modern ML: everything is turned into a vector, and vectors are compared.

Representation learning as a goal

A good representation makes the downstream task easy — ideally linear. The standard measurement is a linear probe: freeze the encoder, train logistic regression on its features, and read the accuracy (5.8 Self-supervised and contrastive learning). Transfer learning (5.4 Transfer learning and training CNNs in practice) reuses representations learned on a big dataset for a small one; pretraining (7.6 Decoder-only: GPT and autoregressive generation, 8.1 The LLM lifecycle and model landscape) is representation learning at the scale of the internet, and fine-tuning adjusts the representation for a task (8.2 Fine-tuning and PEFT). The hidden layers of every network in this phase were already learning representations; autoencoders just make the idea explicit by giving the representation a name and a dimension.

The vocabulary you will carry forward

The same two words mean slightly different things in each row — an encoder always maps inputs to representations, a decoder always produces outputs from representations, but whether the decoder reconstructs, translates or generates depends on the model. When someone says "encoder-only" or "decoder-only" in 7.7 Encoder–decoder: T5, BART and choosing an architecture, this table is what they mean.

In code

class AE(nn.Module):
    def __init__(self, d=784, h=128, k=32):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, k))
        self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d))
    def forward(self, x):
        z = self.enc(x)
        return self.dec(z), z
x_hat, z = ae(x.view(x.size(0), -1))
loss = torch.nn.functional.mse_loss(x_hat, x.view(x.size(0), -1))

Where this shows up later

• PCA as the linear special case — 3.12 Dimensionality reduction: PCA, t-SNE and UMAP; InfoNCE and contrastive representation learning — 4.3 Loss functions catalog, 5.8 Self-supervised and contrastive learning; linear probes and transfer — 5.4 Transfer learning and training CNNs in practice.
• Word embeddings trained by prediction — 6.3 Word embeddings; the encoder–decoder split in seq2seq and transformers — 6.6 Seq2seq, encoder–decoder and beam search, 7.1 Attention Is All You Need: the big picture, 7.7 Encoder–decoder: T5, BART and choosing an architecture; BERT as a masked (denoising) autoencoder — 7.5 Encoder-only: BERT and friends (and sentence embeddings).
• VAEs, latent diffusion's VAE compressor and the denoiser-as-score connection — 8.8 Generative models I: VAEs and GANs, 8.9 Generative models II: Diffusion and flow matching; sparse autoencoders for interpretability — 8.11 Interpretability: what is happening inside; embedding models and vector search — 9.2 Embeddings deep dive, 9.3 Vector search and indexes.