Convolution: the operation and its math

Images are tensors

A grayscale image is a matrix of intensities; a color image is a stack of three matrices (red, green, blue). PyTorch stores a batch of images as a 4-D tensor of shape $(N, C, H, W)$: batch, channels, height, width. Pixel values 0–255 are scaled to floats, typically normalized per channel with the ImageNet mean and standard deviation. Everything a vision model does is a sequence of operations on this tensor, and the first operation is almost always a convolution.

The operation

A kernel (or filter) $\mathbf{K}$ is a small matrix, usually $3\times 3$ or $5\times 5$, of learnable weights. Sliding it over the image and taking a dot product at each position produces a feature map. For a single-channel image $\mathbf{I}$ and a $k\times k$ kernel:

Strictly this is cross-correlation; true convolution flips the kernel, $(\mathbf{I} \ast \mathbf{K})[i,j] = \sum_{u,v} \mathbf{I}[i-u, j-v]\,\mathbf{K}[u,v]$. Since the kernel is learned, a flipped kernel is just another kernel, so deep-learning libraries implement cross-correlation and call it convolution. Each output value is a dot product (1.1 Vectors, dot products and norms) between a patch of the image and the kernel: the kernel is a template, and the feature map records how strongly each location matches it.

With $C_{in}$ input channels and $C_{out}$ filters, the weight tensor has shape $(C_{out}, C_{in}, k, k)$ and each output channel sums over all input channels:

where $s$ is the stride (step between positions) and $p$ the padding (zeros added around the border). The parameter count is $C_{out}\,(C_{in} k^2 + 1)$. A first layer with $C_{in}=3$, $C_{out}=64$, $k=3$ has $64\cdot(27+1) = 1{,}792$ parameters — independent of the image size.

Hand-designed kernels

Before CNNs, people designed kernels by hand. A box blur averages a neighborhood; a Sobel kernel estimates the horizontal gradient:

Worked example. Apply $\mathbf{K}_{\text{sobel-}x}$ to the patch $\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix}$: $(-1)(1)+(1)(3) + (-2)(4)+(2)(6) + (-1)(7)+(1)(9) = 2+4+2 = 8$. Intensity rises by 1 per column, and the kernel's weights sum to $8$ on the right minus left, so the response is positive and proportional to the slope. A CNN learns kernels like these in its first layer — and thousands of more useful ones.

Stride, padding and the output-size formula

For input size $n$, kernel $k$, padding $p$ and stride $s$:

• LeNet-style: $n=32,\ k=5,\ p=0,\ s=1 \Rightarrow 28$.
• "Same" padding keeps the size for stride 1: $p = (k-1)/2$, so $k=3 \Rightarrow p=1$.
• ResNet stem: $n=224,\ k=7,\ p=3,\ s=2 \Rightarrow \lfloor 223/2\rfloor + 1 = 112$. ✓

Stride 2 halves resolution and quarters compute; it is the modern replacement for pooling in many architectures (5.2 CNN building blocks).

Why convolution works: locality, sharing, equivariance

1. Locality. Each output looks at a $k\times k$ neighborhood: the right prior for images, where edges and textures are local.
2. Weight sharing. The same kernel is used at every position. A dense layer from a $224\times224\times3$ image to 64 units needs $\approx 9.6$ million weights; the conv above needs 1,792 — a $5{,}000\times$ saving, and far less overfitting (3.6 Regularization and generalization).
3. Translation equivariance. Shift the input and the feature map shifts by the same amount: $f(T\mathbf{x}) = T f(\mathbf{x})$. A cat detector works in any corner of the image. (Pooling and a final classifier turn equivariance into approximate invariance.)

Intuition. A convolution layer is a dense layer with two constraints: most weights are forced to zero (locality) and the remaining weights are tied together (sharing). Constraints are an inductive bias — knowledge about the data baked into the architecture. Vision Transformers (5.7 Vision Transformers) remove this bias and must learn it from much more data.

Receptive field

The receptive field of a unit is the region of the input that can influence it. Stacking two $3\times3$ layers gives each output a $5\times5$ view with $2\cdot 9 = 18$ weights per channel instead of 25, plus an extra nonlinearity — VGG's key insight (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt). In general, for layer $l$ with kernel $k_l$ and stride $s_l$:

Three stride-1 $3\times3$ layers: $1 \to 3 \to 5 \to 7$. Make the second layer stride 2: $1 \to 3 \to 5 \to 5 + 2\cdot 2 = 9$. Downsampling makes receptive fields grow fast, which is why deep nets with a few stride-2 layers can see the whole image.

Convolution as a sparse linear layer, and its gradients

Unroll each $k\times k\times C_{in}$ patch into a row (the im2col trick) and the convolution becomes one matrix multiplication of a $(H_{out}W_{out}) \times (C_{in}k^2)$ patch matrix with a $(C_{in}k^2)\times C_{out}$ weight matrix. That is how GPUs run it, and it shows convolution is linear: a structured (Toeplitz-like) matrix, nothing more. Backpropagation (4.4 Backpropagation, fully derived) therefore follows the chain rule, with weight sharing adding one twist — the kernel gradient sums contributions from every position:

So the gradient with respect to the kernel is itself a cross-correlation (of the input with the output gradient), and the gradient with respect to the input is a full convolution of $\delta$ with the flipped kernel — exactly a transposed convolution (5.2 CNN building blocks). Autograd does this for you, but knowing it explains why conv backward costs about as much as forward.

FLOPs. Each output value needs $C_{in}k^2$ multiply-adds, so a layer costs $\approx 2\, H_{out} W_{out}\, C_{out}\, C_{in}\, k^2$ FLOPs. The ResNet stem ($112\times112$ output, 64 filters, $7\times7\times3$): $2\cdot 12{,}544 \cdot 64 \cdot 147 \approx 236$ MFLOPs — about 6% of a whole ResNet-50 forward pass (4.1 GFLOPs) in a single layer.

import torch, torch.nn as nn
conv = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)
x = torch.randn(1, 3, 224, 224)
print(conv(x).shape)                     # torch.Size([1, 64, 112, 112])
print(sum(p.numel() for p in conv.parameters()))   # 64*(3*49+1) = 9472

Where this shows up later

• Blocks, pooling, 1×1 and transposed convolutions — 5.2 CNN building blocks; the architectures that stack them — 5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt.
• A ViT's patch embedding is a convolution with kernel = stride = patch size — 5.7 Vision Transformers.
• 1-D causal convolutions build WaveNet in makemore Part 5 — 6.8 Project: character-level language model; the U-Net inside Stable Diffusion is convolutional — 8.9 Generative models II: Diffusion and flow matching.
• FLOP and memory accounting continues for transformers — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings, 4.12 Compute, hardware and scale basics.