Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt

The benchmark that drove everything

ImageNet's ILSVRC challenge: 1.28 million training images, 1,000 classes, scored by top-5 error (the true label is not among the five highest-scoring guesses) and later top-1 accuracy. Before 2012 the best systems used hand-crafted features (SIFT + Fisher vectors) and were stuck around 26% top-5 error. Then:

LeNet-5 (1998) and AlexNet (2012)

LeCun's LeNet established the template: two conv/pool stages then fully connected layers, trained with backprop to read checks. It worked, but compute and data were lacking for anything harder. AlexNet (Krizhevsky, Sutskever, Hinton) was essentially a bigger LeNet made trainable: ReLU instead of tanh (6× faster convergence, 4.2 Activation functions), dropout in the FC layers (4.7 Regularization in deep nets), aggressive data augmentation (crops, flips, color PCA), and training on two GTX 580 GPUs for six days. Five conv layers plus three FC layers; the FC layers held 58 of the 60 million parameters. Halving the error rate overnight started the deep learning era.

VGG (2014): depth with 3×3 kernels

VGG replaced AlexNet's $11\times11$ and $5\times5$ kernels with stacks of $3\times3$ convs. Two stacked $3\times3$ layers have the receptive field of a $5\times5$ with $2\cdot 9C^2 = 18C^2$ weights instead of $25C^2$, and three replace a $7\times7$ with $27C^2$ versus $49C^2$ — fewer parameters, more nonlinearity (5.1 Convolution: the operation and its math). The design is so regular (double channels whenever you halve resolution) that "VGG-style" became shorthand. Costs: 138 M parameters (most in three FC layers) and 15.5 G multiply-adds, which is why VGG survives mainly as a feature extractor for perceptual losses.

GoogLeNet / Inception (2014): go wider, cheaply

An Inception module runs $1\times1$, $3\times3$, $5\times5$ convs and a max-pool in parallel and concatenates the outputs, letting the network pick its own kernel size per layer. The trick that made it affordable is the $1\times1$ bottleneck before each expensive conv (5.2 CNN building blocks): reduce 256 channels to 64, then do the $5\times5$. With global average pooling replacing the FC layers, GoogLeNet needed only 6.8 M parameters — 20× fewer than VGG — at lower error. Auxiliary classifiers injected gradient mid-network (a crutch made unnecessary by BatchNorm, which arrived with Inception-v2 and was later factorized further in v3: $5\times5 \to$ two $3\times3$, $n\times n \to 1\times n$ then $n\times 1$).

ResNet (2015): the degradation problem and its fix

With BatchNorm, people could train 30-layer nets — but a 56-layer plain network had higher training error than a 20-layer one. Not overfitting: the optimizer simply could not find the deeper solution, even though it exists (copy the shallow net, set extra layers to identity). He et al. made identity the default. A residual block computes

so the layers only have to learn the residual $\mathcal{F} = \mathbf{y} - \mathbf{x}$, and "do nothing" is simply $\mathcal{F} \approx 0$ — easy to learn with small weights. Differentiate:

The identity term means the gradient reaching early layers is the sum of a direct path and transformed paths, never just a product of many small Jacobians (6.4 Recurrent neural networks and BPTT shows what that product does to RNNs). Across $n$ blocks the signal takes $2^n$ paths of varying depth; Veit et al. showed trained ResNets behave like ensembles of relatively shallow networks. Practical details: when shapes change, the shortcut uses a $1\times1$ stride-2 conv; the bottleneck block ($1\times1$ reduce → $3\times3$ → $1\times1$ expand) makes ResNet-50/101/152 affordable (ResNet-50: 25.6 M params, 4.1 G multiply-adds); the pre-activation variant (BN → ReLU → conv) trains 1,000-layer nets. Every transformer block is a residual block (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).

Intuition. A residual network is a highway with on-ramps: information and gradients can drive straight through, and each block adds a small correction. Depth stops being a liability.

Siblings: Wide ResNet (fewer, wider layers), ResNeXt (grouped convs — "cardinality"), DenseNet (concatenate all previous features instead of adding), SE-Net (channel attention via squeeze-and-excitation).

MobileNet and EfficientNet: efficiency by design

MobileNet (2017) built an entire network from depthwise-separable blocks (5.2 CNN building blocks) for phones: 4.2 M params, 569 M multiply-adds, 70.6% top-1. MobileNetV2 added inverted residuals (expand → depthwise → project, with the skip on the thin end). EfficientNet (2019) asked how to scale a network properly: depth $d$, width $w$ and input resolution $r$ should grow together. Compound scaling uses one knob $\phi$:

so each increment of $\phi$ roughly doubles FLOPs (compute scales with depth, width², resolution²). Starting from a neural-architecture-searched B0, B1…B7 followed; B7 hit 84.3% top-1 with 8.4× fewer parameters than the previous best.

ConvNeXt (2022): what did transformers actually teach us?

After Vision Transformers (5.7 Vision Transformers) and Swin beat ResNets, Liu et al. asked whether the gains came from attention or from everything else. They modernized a ResNet-50 step by step: train with the DeiT recipe (AdamW, 300 epochs, strong augmentation, stochastic depth) → 78.8% from 76.1%; change stage ratios to 1:1:3:1; a patchify stem ($4\times4$ conv, stride 4); grouped → depthwise convs with wider channels; inverted bottleneck; large $7\times7$ depthwise kernels; GELU for ReLU, fewer activations, LayerNorm for BatchNorm, fewer norm layers; separate downsampling layers. The result, ConvNeXt, matched or beat Swin at every scale (ConvNeXt-T 82.1% vs Swin-T 81.3%) with pure convolutions. The lesson: much of the "transformer advantage" was recipe and macro-design, and the architecture debate matters less than the training pipeline.

Reading an architecture table

Papers describe networks as tables: stage name, output size, block definition, repeat count. Reproduce them: compute each output size with the formula from 5.1 Convolution: the operation and its math, count parameters per block (5.2 CNN building blocks), and compare the total to the paper. If the numbers match you understand the network. Tools: torchinfo.summary, fvcore for FLOPs, and timm.list_models() for hundreds of implementations.

Where this shows up later

• Fine-tuning these backbones with modern recipes — 5.4 Transfer learning and training CNNs in practice; backbones with FPN inside detectors — 5.5 Object detection.
• Residual connections and pre-norm in the transformer block — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; normalization choices — 4.6 Initialization and normalization.
• ViT and Swin, the models ConvNeXt was answering — 5.7 Vision Transformers; scaling laws as the LLM version of compound scaling — 7.9 Scaling laws and compute-optimal training.