CNN building blocks

The anatomy of a CNN

Almost every CNN has three parts. A stem quickly reduces resolution (e.g., a $7\times7$ stride-2 conv plus a stride-2 pool: $224 \to 56$). A sequence of stages each halve the spatial size while roughly doubling the channels (56→28→14→7, channels 64→128→256→512), keeping the compute per stage roughly constant since cost scales with $H W C_{in} C_{out}$. A head collapses the final $7\times7\times C$ map with global average pooling into a $C$-vector and applies a linear classifier. Total downsampling of $32\times$ (the output stride) is the classification default; dense-prediction tasks reduce it (5.6 Segmentation).

Pooling

Max pooling with a $2\times2$ window and stride 2 keeps the strongest response in each window and halves both spatial dimensions (output size follows the usual formula with $k=s=2$, $p=0$). It has no parameters, adds a little translation invariance (a feature shifted by one pixel still wins its window) and quarters the compute of everything downstream. Average pooling blurs instead of selecting. Global average pooling (GAP) averages each channel over all positions, turning a $7\times7\times2048$ map into 2,048 numbers; introduced by Network-in-Network and used by GoogLeNet and ResNet, it removed the giant fully connected layers that held 90% of AlexNet's and VGG's parameters. Many modern nets drop pooling entirely and downsample with stride-2 convolutions, which learn how to downsample ("all convolutional" networks, ResNet stages).

1×1 convolutions

A $1\times1$ convolution applies the same linear map to the channel vector at every pixel:

It mixes channels but not positions — a per-pixel fully connected layer with $C_{in}C_{out}$ parameters. Uses: shrinking channels before an expensive $3\times3$ (the bottleneck of ResNet-50, Inception), expanding them after, and adding nonlinearity cheaply. A transformer's position-wise feed-forward network is exactly this idea applied to tokens (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).

Depthwise-separable convolutions

A standard $k\times k$ conv costs $H W\, C_{in} C_{out} k^2$ multiply-adds because every filter mixes all channels and all positions at once. Factor it into two steps: a depthwise conv (one $k\times k$ filter per input channel, no mixing across channels) costing $H W\, C_{in} k^2$, then a pointwise $1\times1$ conv mixing channels at $H W\, C_{in} C_{out}$. The ratio of separable to standard cost is

For $k=3$ and $C_{out}=256$: $1/256 + 1/9 \approx 0.115$, about $8.7\times$ cheaper, with a similar saving in parameters. This is the engine of MobileNet and Xception (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt) and of the large-kernel ($7\times7$) depthwise layers in ConvNeXt — once mixing is cheap, big spatial kernels become affordable.

Dilated (atrous) convolutions

Insert $d-1$ zeros between kernel taps. A $k\times k$ kernel with dilation $d$ covers $k + (k-1)(d-1)$ pixels while still using $k^2$ weights. Stacking dilations 1, 2, 4, 8 grows the receptive field exponentially without any downsampling — WaveNet uses this in 1-D (6.8 Project: character-level language model) and DeepLab in 2-D for segmentation, where you want context and full resolution (5.6 Segmentation).

Transposed convolutions: learning to upsample

Segmentation, autoencoders (4.10 Autoencoders, embeddings and representation learning) and GAN generators (8.8 Generative models I: VAEs and GANs) must go from a small map back to a large one. A transposed convolution (sometimes misnamed "deconvolution") is the adjoint of a strided convolution: each input value is multiplied by the kernel and scattered into an output region, with overlaps summed. Its output size inverts the conv formula:

For $n_{in}=7,\ k=4,\ s=2,\ p=1$: $(6)(2) - 2 + 4 = 14$. When $k$ is not divisible by $s$ the overlaps are uneven, producing checkerboard artifacts; the standard fix is nearest/bilinear upsampling followed by an ordinary conv, or pixel shuffle. Recall from 5.1 Convolution: the operation and its math that the gradient of a convolution with respect to its input is a transposed convolution — the two operations are a matched pair.

Normalization and activation placement

The canonical block is conv → BatchNorm → ReLU (4.6 Initialization and normalization, 4.2 Activation functions). At inference BatchNorm folds into the preceding conv's weights and bias, so it is free. Modern blocks (ConvNeXt) use LayerNorm and GELU, following transformers, and place a single activation per block rather than one after every conv.

Worked example: counting a LeNet-style network

Input $1\times28\times28$ (MNIST). Layers: conv $5\times5$, 6 filters, padding 2 → ReLU → maxpool 2 → conv $5\times5$, 16 filters → ReLU → maxpool 2 → flatten → FC 120 → FC 84 → FC 10.

Two lessons jump out: the fully connected layers hold 96% of the parameters but only 14% of the compute, while the convolutions are the reverse. This is why GAP replaced FC layers (parameters) and why architecture work focuses on the conv stages (compute). Use the same accounting on any network table — 4.12 Compute, hardware and scale basics extends it to memory and transformers.

Where this shows up later

• Bottlenecks, GAP, depthwise convs and large kernels in the architecture timeline — 5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt.
• Upsampling and skip connections in U-Net and Mask R-CNN — 5.6 Segmentation; feature pyramids in detection — 5.5 Object detection.
• The transformer FFN as a 1×1 convolution over tokens — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; dilated causal convs in WaveNet — 6.8 Project: character-level language model; transposed convs in GAN generators — 8.8 Generative models I: VAEs and GANs.