Segmentation

Three flavors of segmentation

Semantic segmentation assigns every pixel a class (road, car, sky) without distinguishing instances. Instance segmentation finds each object and its mask (car 1, car 2) but ignores amorphous "stuff" like road. Panoptic segmentation does both: every pixel gets a class and, for countable "things", an instance id. Each flavor has its benchmarks (Cityscapes, ADE20K, COCO-Stuff/Panoptic) and losses, but they share one engineering problem: producing output at full image resolution from features that a classifier would have downsampled 32×.

Fully convolutional networks

FCN (2015) made the first move: take a classification CNN, replace the fully connected layers with $1\times1$ convolutions (5.2 CNN building blocks) so the network outputs a coarse class-score map, then upsample it to the input size with bilinear interpolation or learned transposed convolutions. Upsampling a 32×-downsampled map alone gives blobby boundaries, so FCN-16s and FCN-8s add skip connections: fuse predictions from earlier, higher-resolution layers. The idea generalizes: deep layers know what, shallow layers know where; combine them.

U-Net: the encoder–decoder with skips

U-Net (2015, Ronneberger et al.) made the combination systematic. A contracting encoder (conv–conv–pool, channels doubling) is mirrored by an expanding decoder (upsample–conv–conv, channels halving), and at every resolution the encoder's feature map is concatenated onto the decoder's — the "U" shape with horizontal skips. The decoder thus gets semantic context from below and precise localization from the side; gradients also flow through short paths, which is why U-Nets train quickly even from a few dozen annotated biomedical images with heavy elastic augmentation. Modern variants pad convolutions so output size equals input size, swap the encoder for a pretrained ResNet/ConvNeXt (via libraries like segmentation_models.pytorch), and add attention or residual blocks. The same architecture, with time and text conditioning injected, is the noise predictor of DDPM and Stable Diffusion (8.9 Generative models II: Diffusion and flow matching) — the encoder/decoder vocabulary from 4.10 Autoencoders, embeddings and representation learning in its most literal form.

DeepLab took another route to resolution: keep the feature map at output stride 8 or 16 using dilated convolutions (5.2 CNN building blocks) and gather multi-scale context with Atrous Spatial Pyramid Pooling (parallel dilated convs with different rates). Transformer-based segmenters (SegFormer, Mask2Former) now lead benchmarks, but U-Net remains the default for medical and small-data problems.

Losses for masks

Per-pixel cross-entropy (4.3 Loss functions catalog) is the natural loss, but masks are often tiny: a tumor may be 1% of the pixels, so a model predicting "background everywhere" scores 99% pixel accuracy and a low cross-entropy. Overlap-based losses optimize what we actually measure. The Dice coefficient between predicted mask $A$ and ground truth $B$:

Worked: $|A| = |B| = 100$ pixels with 60 overlapping. IoU $= 60/(100+100-60) = 60/140 \approx 0.429$; Dice $= 120/200 = 0.6$; check: $2(0.429)/1.429 = 0.6$. ✓ Dice is always at least IoU and less harsh on partial overlaps.

To train with it, make it differentiable using predicted probabilities $p_i \in [0,1]$ and binary targets $g_i$ — the soft Dice loss:

with $\epsilon$ for numerical stability when a class is absent. Because it is a ratio over the whole mask, it is insensitive to how many background pixels exist — exactly the imbalance fix we wanted. Its gradients can be noisy for small structures, so the standard recipe is BCE + Dice (or focal + Dice, 5.5 Object detection); the Lovász-softmax loss optimizes a convex surrogate of IoU directly; boundary losses sharpen edges.

Metrics

Pixel accuracy is nearly meaningless under imbalance. Report mean IoU over classes (mIoU, the Cityscapes/ADE20K standard) or per-class Dice (the medical standard). Panoptic quality combines segmentation and recognition: $\mathrm{PQ} = \underbrace{\frac{\sum_{(p,g)\in TP}\mathrm{IoU}(p,g)}{|TP|}}_{\text{SQ}} \times \underbrace{\frac{|TP|}{|TP| + \tfrac{1}{2}|FP| + \tfrac{1}{2}|FN|}}_{\text{RQ}}$, with matches defined by IoU $> 0.5$.

Instance segmentation: Mask R-CNN

Mask R-CNN (2017) adds a third branch to Faster R-CNN (5.5 Object detection): for each detected region, a small FCN predicts a $28\times28$ binary mask per class, using a sigmoid rather than a softmax across classes so mask prediction does not compete with classification. Its enabling detail is RoIAlign: RoI pooling snapped region boundaries to the feature grid (quantization errors of several input pixels), which barely matters for boxes but ruins masks; RoIAlign samples features at exact fractional positions with bilinear interpolation, improving mask AP by 10–50% relative. Runs at about 5 fps; extended to keypoints by predicting one-hot masks per joint.

Segment Anything: promptable segmentation

SAM (2023) reframed segmentation as a promptable task, like a language model for pixels. A heavy ViT-H image encoder (5.7 Vision Transformers, pretrained with MAE, 5.8 Self-supervised and contrastive learning) embeds the image once; a lightweight prompt encoder embeds points, boxes or rough masks (and text via CLIP); a small mask decoder (two-way attention) outputs several candidate masks with confidence scores in milliseconds — interactive segmentation in the browser. The training data came from a data engine: annotate with model assistance, retrain, repeat, producing SA-1B with 11 million images and 1.1 billion masks. SAM transfers zero-shot to new domains, accelerates annotation enormously, and SAM 2 (2024) extended it to video with a streaming memory. It does not know class names — pair it with a detector or a vision-language model to name what it segments (8.10 Multimodal models).

In practice

Semantic: torchvision.models.segmentation (DeepLabV3, FCN), segmentation_models.pytorch (U-Net with 500+ encoders), Hugging Face transformers (SegFormer, Mask2Former). Instance: Mask R-CNN in torchvision, YOLO-seg in Ultralytics. Promptable: SAM/SAM 2 via transformers or the official repo. Datasets store masks as PNG label maps (semantic) or COCO RLE polygons (instance). Augment masks and images with the same geometric transform (Albumentations does this), never interpolate label maps (use nearest-neighbour), and evaluate per class — the rare classes are where models fail.

Where this shows up later

• U-Net as the denoiser in diffusion models — 8.9 Generative models II: Diffusion and flow matching; ViT encoders in SAM — 5.7 Vision Transformers; MAE pretraining — 5.8 Self-supervised and contrastive learning.
• Promptable, multimodal segmentation and open-vocabulary detection — 8.10 Multimodal models.
• Dice/IoU as overlap metrics connect to IoU in detection — 5.5 Object detection and to retrieval metrics — 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics.