Generative models I: VAEs and GANs

Generative modeling: learn $p(\mathbf{x})$

A discriminative model learns $p(y\mid\mathbf{x})$; a generative model learns the data distribution $p(\mathbf{x})$ itself, so it can sample new data, score how likely an input is, and often represent data in a useful latent space. The main families: autoregressive models (7.6 Decoder-only: GPT and autoregressive generation — exact likelihood, slow sampling), latent-variable models (VAEs), adversarial models (GANs), and diffusion/flow models (8.9 Generative models II: Diffusion and flow matching). Each trades off sample quality, diversity/coverage, likelihood evaluation and sampling speed.

Latent variables and the intractable integral

Assume data is produced by first drawing a latent code $\mathbf{z}\in\mathbb{R}^{d}$ from a simple prior $p(\mathbf{z}) = \mathcal{N}(\mathbf{0},\mathbf{I})$, then decoding it with a neural network: $p_\theta(\mathbf{x}\mid\mathbf{z})$. The marginal likelihood is

an integral over all codes — intractable for a nonlinear decoder, as is the posterior $p_\theta(\mathbf{z}\mid\mathbf{x})$ we would need to infer codes. Variational inference replaces the true posterior with a learned encoder $q_\phi(\mathbf{z}\mid\mathbf{x})$, typically a diagonal Gaussian $\mathcal{N}(\boldsymbol{\mu}_\phi(\mathbf{x}), \operatorname{diag}\boldsymbol{\sigma}^2_\phi(\mathbf{x}))$.

The ELBO, derived

Multiply and divide by $q_\phi$ inside the log and apply Jensen's inequality ($\log$ is concave, so $\log\mathbb{E}[\cdot]\ge\mathbb{E}[\log\cdot]$):

This is the evidence lower bound (ELBO). The gap is exactly $\mathbb{D}_{\mathrm{KL}}(q_\phi(\mathbf{z}\mid\mathbf{x})\,\|\,p_\theta(\mathbf{z}\mid\mathbf{x}))$ — you can verify $\log p_\theta(\mathbf{x}) = \text{ELBO} + \mathbb{D}_{\mathrm{KL}}(q_\phi\|p_\theta(\mathbf{z}\mid\mathbf{x}))$ by expanding the posterior with Bayes' rule (1.7 Probability II: Bayes, independence, MLE and MAP). Maximizing the ELBO therefore raises the likelihood and pulls the encoder toward the true posterior.

The two terms have plain meanings. With a Gaussian decoder of fixed variance, the reconstruction term is a negative squared error; with a Bernoulli decoder it is a negative binary cross-entropy (4.3 Loss functions catalog). The KL between the diagonal-Gaussian encoder and the standard-normal prior has a closed form:

which is zero only when the encoder outputs the prior exactly — it keeps the latent space smooth and centered so that sampling $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ at test time lands on codes the decoder understands.

The reparameterization trick

The reconstruction term needs gradients through a random sample $\mathbf{z}\sim q_\phi$. Write the sample as a deterministic function of the parameters and fresh noise:

Now the randomness is in $\boldsymbol{\epsilon}$, which does not depend on $\phi$, and gradients flow through $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ with far lower variance than score-function (REINFORCE, 4.11 Reinforcement learning essentials) estimators. The whole model trains end to end with a single Monte Carlo sample per example.

mu, logvar = encoder(x)
z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu)   # reparameterize
x_hat = decoder(z)
recon = F.binary_cross_entropy_with_logits(x_hat, x, reduction="sum")
kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
loss = recon + kl                                         # negative ELBO

VAE pathologies and variants

• Blurry samples: a pixel-wise Gaussian likelihood rewards averaging over plausible outputs.
• Posterior collapse: a powerful decoder ignores $\mathbf{z}$, the KL term goes to zero and the latent space carries nothing; remedies include KL annealing and weaker decoders.
• β-VAE weights the KL term by $\beta>1$ to encourage disentangled factors at the cost of reconstruction.
• VQ-VAE replaces the Gaussian latent with a learned codebook of discrete codes, giving tokens that autoregressive models can predict — the route taken by DALL·E 1, modern image tokenizers and neural audio codecs (8.10 Multimodal models).
• Most importantly, VAEs are the compressors inside latent diffusion (8.9 Generative models II: Diffusion and flow matching): Stable Diffusion's VAE maps a 512×512 image to a 64×64×4 latent and back.

GANs: a two-player game

A generator $G$ maps noise $\mathbf{z}$ to a sample; a discriminator $D$ outputs the probability that an input is real. They play

Optimal discriminator. For a fixed $G$ with sample distribution $p_g$, maximize pointwise $a\log D + b\log(1-D)$ with $a=p_{\text{data}}(\mathbf{x})$, $b=p_g(\mathbf{x})$; setting the derivative to zero gives

Substituting back, the generator's objective becomes $C(G) = -\log 4 + 2\,\mathbb{D}_{\mathrm{JS}}(p_{\text{data}}\,\|\,p_g)$: at the discriminator's optimum, the generator minimizes the Jensen–Shannon divergence to the data, with global minimum $p_g = p_{\text{data}}$. In practice $G$ maximizes $\log D(G(\mathbf{z}))$ instead of minimizing $\log(1-D(G(\mathbf{z})))$ — the non-saturating loss — because the original saturates when $D$ confidently rejects early samples.

Why GANs are hard, and the fixes

• Mode collapse: $G$ produces a few convincing modes and ignores the rest of the data.
• Vanishing gradients: when real and generated distributions have disjoint support, JS divergence saturates and $D$ gives $G$ no useful signal. WGAN replaces it with the Wasserstein (earth-mover) distance, estimated by a critic constrained to be 1-Lipschitz (via gradient penalty or spectral normalization), restoring smooth gradients.
• Architecture and tricks: DCGAN's convolutional guidelines, two-time-scale updates, feature matching; StyleGAN's mapping network and per-layer style modulation gave photorealistic faces and a controllable latent space; CycleGAN learned unpaired image-to-image translation with cycle-consistency losses.
• Evaluation: Fréchet Inception Distance (FID) compares Gaussian fits of Inception features of real and generated images; precision/recall metrics separate quality from coverage.

2026: where VAEs and GANs live now

Diffusion and flow models (8.9 Generative models II: Diffusion and flow matching) won image and video generation on stability, diversity and scalability. But VAEs are everywhere as latent compressors and tokenizers, and adversarial losses survive as perceptual/adversarial terms in those compressors, in super-resolution, and in adversarial diffusion distillation that turns 50-step samplers into 1–4 step generators. Knowing both families is still essential.

Where this shows up later

• The ELBO and the KL-to-prior reappear, term by term, in diffusion models — 8.9 Generative models II: Diffusion and flow matching.
• VQ-VAE codes become the image and audio tokens of multimodal LLMs — 8.10 Multimodal models.
• Encoder/decoder vocabulary and autoencoders — 4.10 Autoencoders, embeddings and representation learning; the same KL divergence regularizes RLHF — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI.