Self-supervised and contrastive learning

Supervision from the data itself

A pretext task is a problem whose labels come for free: predict the rotation applied to an image, solve a jigsaw of its patches, colorize a grayscale version, inpaint a missing region. Solving it well requires understanding content, so the learned features transfer. Early pretext tasks lagged supervised pretraining; two families closed and then reversed the gap — contrastive/joint-embedding methods (two views of the same image should have similar embeddings) and masked image modeling (reconstruct hidden patches). The philosophy matches language modeling (7.6 Decoder-only: GPT and autoregressive generation): predict part of the data from the rest, at scale.

Contrastive learning and InfoNCE

Take a mini-batch of $N$ images and create two random augmentations of each (random crop + resize, flip, color distortion, blur) — $2N$ views. An encoder $f$ (ResNet or ViT) maps each view to a representation $\mathbf{h}$, and a small projection head $g$ maps it to $\mathbf{z} = g(\mathbf{h})$. Define cosine similarity $s_{ij} = \mathbf{z}_i^\top\mathbf{z}_j / (\lVert\mathbf{z}_i\rVert\,\lVert\mathbf{z}_j\rVert)$ (1.1 Vectors, dot products and norms). For a positive pair $(i, j)$ — the two views of one image — the NT-Xent (normalized temperature-scaled cross-entropy) loss is

Read it as a softmax classifier: among the $2N-1$ other views, pick the one that came from the same image. The other $2N-2$ views are negatives. This is a cross-entropy loss (1.9 Information theory: entropy, cross-entropy and KL divergence) over a $(2N-1)$-way classification whose "logits" are scaled similarities, so everything you know about softmax and cross-entropy applies. It is also an estimator of mutual information: van den Oord et al. showed $I(\mathbf{z}_i;\mathbf{z}_j) \ge \log(2N-1) - \mathcal{L}_{\text{InfoNCE}}$, which is why more negatives (larger batches) help — the bound gets tighter.

Temperature. $\tau$ sharpens or flattens the distribution. Worked: one positive with $s = 0.8$, two negatives with $s = 0.2$ and $0.1$.

• $\tau = 1$: softmax of $(0.8, 0.2, 0.1)$ gives $p_{\text{pos}} = e^{0.8}/(e^{0.8}+e^{0.2}+e^{0.1}) \approx 2.226/4.552 \approx 0.489$, loss $\approx 0.715$.
• $\tau = 0.1$: logits $(8, 2, 1)$ give $p_{\text{pos}} \approx 2981/2991 \approx 0.9966$, loss $\approx 0.0034$.

Small $\tau$ makes the loss focus on hard negatives (the gradient weight of negative $k$ is $\propto \exp(s_{ik}/\tau)$), which drives uniformity on the sphere; too small and training becomes unstable or over-emphasizes false negatives (two different photos of dogs). SimCLR used $\tau = 0.1$–$0.5$; CLIP learns $\tau$.

import torch, torch.nn.functional as F
def nt_xent(z1, z2, tau=0.1):
    z = F.normalize(torch.cat([z1, z2]), dim=1)        # (2N, d), unit vectors
    sim = z @ z.T / tau                                  # cosine similarities / tau
    sim.fill_diagonal_(float("-inf"))                    # exclude self-similarity
    n = z1.shape[0]
    targets = torch.cat([torch.arange(n, 2*n), torch.arange(0, n)])  # index of each positive
    return F.cross_entropy(sim, targets)                 # softmax over 2N-1 candidates

SimCLR and MoCo

SimCLR (2020) is exactly the recipe above and found the details that matter: composition of crop and color distortion is essential (otherwise the model matches color histograms), a nonlinear projection head helps but is discarded after training (the representation before it transfers better), and large batches (4,096–8,192) with many negatives help. Linear probe on ImageNet: 69.3% with ResNet-50, 76.5% with a 4×-wide ResNet-50, matching supervised accuracy. MoCo removed the big-batch requirement: negatives come from a queue of past embeddings produced by a momentum encoder whose weights are an exponential moving average of the online encoder, $\theta_k \leftarrow m\theta_k + (1-m)\theta_q$ with $m = 0.999$, keeping the queue consistent. MoCo v3 adapted the idea to ViTs.

Non-contrastive methods: no negatives, no collapse?

Why not just pull positives together? Because the trivial solution — map every image to the same vector — gives zero loss (collapse). Negatives prevent it in contrastive methods; BYOL (2020) showed you can avoid it differently: an online network with an extra predictor head learns to predict the output of a slowly-moving target network (EMA weights) on the other view, with a stop-gradient on the target. The asymmetry (predictor + EMA + stop-gradient) empirically prevents collapse; SimSiam showed even the EMA is optional if the stop-gradient stays. DINO (2021) frames this as self-distillation: a student ViT matches the softmax output of an EMA teacher over the other view, with centering (subtract a running mean so no single dimension dominates) and sharpening (low teacher temperature) jointly preventing collapse. Its striking discovery: the self-attention maps of a DINO ViT's [CLS] token segment the main object without any labels. DINOv2 (2023) scaled this with a curated 142 M-image dataset, a ViT-g teacher distilled into smaller models, and became the default frozen backbone — its features give state-of-the-art linear probes for classification, depth estimation and segmentation without fine-tuning.

Masked image modeling

MAE (2021) ports BERT to pixels with two twists. Mask a large fraction (75%) of patches — images are redundant, so a small mask would be trivially solved by interpolation. Run the encoder on the visible patches only (so it processes 25% of tokens: 3–4× cheaper), then a lightweight decoder takes encoded patches plus mask tokens and reconstructs normalized pixel values with an MSE loss on masked patches. MAE pretraining made ViT-H reach 87.8% ImageNet top-1 with ImageNet-1K data only and transfers excellently when fine-tuned, though its linear-probe accuracy is lower than DINO's. Rule of thumb: masked modeling → best after fine-tuning; joint-embedding → best frozen (linear probe / kNN). Many modern recipes combine both (DINOv2 adds an iBOT-style masked objective).

Evaluating representations

• Linear probe: freeze the encoder, train logistic regression (3.3 Logistic regression and softmax classification) on features. Measures linear separability.
• kNN classifier: no training at all (3.4 k-Nearest Neighbors); DINO reports it routinely.
• Fine-tuning: the ceiling, but conflates pretraining quality with the recipe.
• Transfer to detection/segmentation and robustness sets. The same protocol evaluates text embedding models in 9.2 Embeddings deep dive (MTEB is a linear-probe-and-retrieval suite).

Where this shows up later

• CLIP applies InfoNCE across images and captions — 8.10 Multimodal models; text embedding models for RAG train with in-batch negatives — 9.2 Embeddings deep dive.
• Masked language modeling (BERT) is MIM's older sibling — 7.5 Encoder-only: BERT and friends (and sentence embeddings); the ViT encoders here are reused by SAM — 5.6 Segmentation.
• Interpretability of what emerges without labels — 8.11 Interpretability: what is happening inside; representation learning foundations — 4.10 Autoencoders, embeddings and representation learning.