Vision Transformers

From pixels to tokens

A transformer consumes a sequence of vectors. ViT turns an image into one by patchifying: split an $H\times W\times C$ image into non-overlapping $P\times P$ patches, flatten each into a vector of length $P^2 C$, and project it linearly to the model width $D$:

For ViT-B/16 at $224\times224$: $P = 16$, so $N = (224/16)^2 = 196$ patches, each a $16\cdot16\cdot3 = 768$-dimensional vector, projected to $D = 768$. The projection $\mathbf{E}$ is exactly a convolution with kernel size 16 and stride 16 (5.1 Convolution: the operation and its math) — $768\times768 + 768 \approx 590$K parameters. A learnable [CLS] token is prepended (197 tokens total) and a learned position embedding is added so the model knows where each patch came from (attention itself is permutation-equivariant; see 7.3 Positional information: sinusoidal, learned, RoPE, ALiBi).

Just enough attention

Each transformer layer lets every token gather information from every other token. Tokens are projected to queries, keys and values; attention weights are the softmax of scaled query–key dot products (1.1 Vectors, dot products and norms), and each output is the weighted average of values:

Several heads do this in parallel in different subspaces, a position-wise MLP follows, and both are wrapped in residual connections (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt) with LayerNorm in front (pre-norm). That is the entire ViT encoder — identical to the text transformer encoder of 7.1 Attention Is All You Need: the big picture, which is why one architecture now serves both modalities. ViT-B/16: 12 layers, $D=768$, 12 heads, MLP width 3072, 86 M parameters, 17.6 GFLOPs at $224^2$. The cost of attention is $O(N^2 D)$: fine for 196 tokens, painful at $1024^2$ images (4,096 tokens with $P=16$) — the motivation for Swin and for patch sizes of 14–32.

Intuition. A convolution lets a pixel see its $3\times3$ neighbors. Attention lets a patch see every other patch from layer one, with weights it chooses per image. More flexible, but it has to learn that nearby patches usually matter most.

Inductive bias versus data

CNNs hard-wire locality, translation equivariance and a multi-scale hierarchy (5.1 Convolution: the operation and its math). ViT hard-wires almost nothing beyond the patch grid: position embeddings are learned, and only the MLP is local. The original paper's central finding follows: trained on ImageNet-1K alone (1.3 M images), ViT trails ResNets; pretrained on ImageNet-21K (14 M) it matches them; on JFT-300M it wins (ViT-H/14 at 88.55% top-1). With enough data, learned biases beat designed ones. Inspecting trained models shows why this can work: some heads in the first layer attend locally (learning convolution-like behavior) while others are already global, and "Do Vision Transformers See Like CNNs?" found ViTs keep more uniform representations across depth and integrate global information much earlier.

DeiT (2021) closed the data gap without extra images: a heavy augmentation and regularization recipe (RandAugment, mixup, CutMix, stochastic depth, repeated augmentation) plus distillation through a dedicated distillation token that learns from a strong CNN teacher (RegNetY). DeiT-B reached 81.8% top-1 (83.4% distilled) on ImageNet-1K alone — the recipe that ConvNeXt later borrowed to modernize ResNets (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt).

Hierarchical ViTs: Swin

Classification tolerates a single $14\times14$ grid, but detection and segmentation (5.5 Object detection, 5.6 Segmentation) want multi-scale features and high resolution. Swin Transformer restores a CNN-like pyramid: start with $4\times4$ patches, compute attention only inside non-overlapping $7\times7$ windows (cost linear in image size), shift the windows by half a window in alternate layers so information crosses window borders, and merge $2\times2$ patches between stages to halve resolution and double channels. The result is a drop-in backbone for FPN-based detectors; Swin-L reached 87.3% ImageNet top-1 and 58.7 box AP on COCO at release. Relative position biases replace absolute embeddings.

Position information in two dimensions

Options: learned 1-D embeddings per patch index (ViT; they still learn a 2-D structure), 2-D sinusoidal embeddings, relative position biases added to attention logits (Swin), and 2-D rotary embeddings in recent models (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi). When fine-tuning at a higher resolution than pretraining, interpolate the position embeddings to the new grid — a standard trick that works surprisingly well.

Choosing in 2026

The architecture war ended in a draw on ImageNet (ConvNeXt vs Swin), but pretraining tilted the field: the strongest general-purpose vision encoders are ViTs because they scale smoothly with data and compute and share machinery with language models, making multimodal fusion natural.

import timm, torch
vit = timm.create_model("vit_base_patch16_224", pretrained=True)
x = torch.randn(1, 3, 224, 224)
tokens = vit.patch_embed(x)              # (1, 196, 768): 196 patch tokens of width 768
print(tokens.shape, sum(p.numel() for p in vit.parameters()) / 1e6, "M params")

Where this shows up later

• The full transformer block and attention math — 7.1 Attention Is All You Need: the big picture, 7.2 Scaled dot-product and multi-head attention, fully derived; positional encodings — 7.3 Positional information: sinusoidal, learned, RoPE, ALiBi.
• Self-supervised ViTs (DINO, MAE) and contrastive pretraining — 5.8 Self-supervised and contrastive learning; CLIP and vision-language models — 8.10 Multimodal models.
• SAM's image encoder is a ViT — 5.6 Segmentation; Vision Transformers as diffusion backbones (DiT) — 8.9 Generative models II: Diffusion and flow matching.