Transfer learning and training CNNs in practice

Why pretrained features transfer

Yosinski et al. (2014) split an ImageNet network and measured how well layers transfer to a different task. First-layer filters are Gabor-like edge and color detectors — general for any natural image. Middle layers detect textures and parts; the last layers are specific to the training classes. Transferability decreases with depth, but even specific layers help as initialization: fine-tuning from pretrained weights beats random initialization on almost every dataset, and the gap grows as your dataset shrinks. In 2026 the starting points are better than ever: ImageNet-21K and JFT-pretrained ConvNeXts and ViTs, and self-supervised or language-supervised foundation encoders (DINOv2, CLIP, SigLIP — 5.8 Self-supervised and contrastive learning, 8.10 Multimodal models) whose features work well frozen.

Two regimes

Feature extraction is fast (one forward pass per image, then a tiny classifier) and nearly impossible to overfit. Full fine-tuning gives the best accuracy when data allows. A middle path: unfreeze progressively from the top (gradual unfreezing) and use discriminative learning rates — small LRs for early layers (keep the general features), larger for the head.

The fine-tuning recipe (2026 defaults)

1. Backbone: timm offers 1,000+ pretrained weights; good first picks are convnext_tiny or resnet50 for speed, vit_base_patch16_224 or a DINOv2 ViT for quality.
2. Head: replace the classifier with a fresh linear layer of num_classes outputs.
3. Preprocessing: resize/crop to the model's training resolution and normalize with the pretraining mean/std (timm stores them in the model config). Mismatched normalization is the classic silent bug.
4. Optimizer: AdamW, weight decay 0.05, head LR $10^{-3}$, backbone LR $10^{-4}$ (10× lower); or SGD + momentum 0.9 for ResNets.
5. Schedule: 1–3 warmup epochs then cosine decay (4.5 Optimizers and learning-rate schedules); 10–30 epochs is typical for fine-tuning.
6. Regularization: label smoothing 0.1, stochastic depth 0.1 for deep nets, dropout only in the head, EMA of weights for a free boost.
7. Precision: bf16/fp16 autocast halves memory and doubles throughput (4.9 PyTorch II: real training pipelines).

import timm, torch, torch.nn as nn
model = timm.create_model("convnext_tiny", pretrained=True, num_classes=NUM_CLASSES)
cfg = timm.data.resolve_data_config({}, model=model)        # mean/std/input size
train_tf = timm.data.create_transform(**cfg, is_training=True, auto_augment="rand-m9-mstd0.5")
val_tf   = timm.data.create_transform(**cfg, is_training=False)

head_params = [p for n, p in model.named_parameters() if n.startswith("head")]
body_params = [p for n, p in model.named_parameters() if not n.startswith("head")]
opt = torch.optim.AdamW([{"params": body_params, "lr": 1e-4},
                         {"params": head_params, "lr": 1e-3}], weight_decay=0.05)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=[1e-4, 1e-3],
                                            total_steps=EPOCHS * len(train_loader), pct_start=0.1)
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
scaler = torch.amp.GradScaler()

for epoch in range(EPOCHS):
    model.train()
    for x, y in train_loader:
        x, y = x.cuda(), y.cuda()
        with torch.autocast("cuda", dtype=torch.bfloat16):
            loss = loss_fn(model(x), y)
        opt.zero_grad(set_to_none=True)
        scaler.scale(loss).backward()
        scaler.step(opt); scaler.update(); sched.step()

Data augmentation: the strongest regularizer in vision

Augmentation multiplies your dataset with label-preserving transformations and teaches invariances (4.7 Regularization in deep nets). Standard: random resized crop (scale 0.08–1 for ImageNet; gentler, 0.5–1, for small datasets), horizontal flip, color jitter. Automated policies: RandAugment / TrivialAugment apply random operations at random magnitudes and are strong defaults. Mixing methods regularize the decision boundary:

CutMix pastes a patch from $\mathbf{x}_j$ into $\mathbf{x}_i$ with $\lambda$ equal to the area fraction. Random erasing hides rectangles. Think before you augment: vertical flips break digits and text, hue shifts break species identification by color, and crops that remove the lesion destroy the label in medical images. At test time, averaging predictions over a few flips/crops (test-time augmentation) adds a reliable half-point.

Imbalance and tiny datasets

For imbalanced classes use class-weighted cross-entropy or a weighted sampler, and tune the decision threshold on validation rather than trusting 0.5 (3.5 Evaluating models: metrics, validation and the bias–variance trade-off). Focal loss (5.5 Object detection) helps when the easy majority dominates. With very few labels per class (5–50), skip training a CNN head: extract frozen embeddings from a strong backbone and fit logistic regression or kNN (3.4 k-Nearest Neighbors) on them — this often beats fine-tuning and is trivially reproducible. If you have lots of unlabeled domain images, self-supervised pretraining on them (5.8 Self-supervised and contrastive learning) is the next lever.

Resolution, batch size and learning rate

Train and test resolution interact: networks trained with random-resized-crop see objects larger than at test time, so testing at a slightly higher resolution (or fine-tuning the head at test resolution, "FixRes") improves accuracy. Progressive resizing — train small, finish at full size — saves compute. When you change batch size, scale the learning rate linearly (4.5 Optimizers and learning-rate schedules) and check memory with the arithmetic from 4.12 Compute, hardware and scale basics.

Evaluate, then debug

Report accuracy and per-class precision/recall; plot the confusion matrix and look at the most confused pairs; inspect the highest-loss training images (fast.ai's plot_top_losses) — you will find label errors and near-duplicates. Check calibration (reliability diagram, 5.9 Interpretability, robustness and adversarial examples). When validation lags, the usual fixes in order: get more or cleaner data, stronger augmentation, a better backbone, more epochs — architecture tweaks come last.

When to train from scratch

He et al. ("Rethinking ImageNet Pre-training", 2018) showed that with enough iterations a from-scratch detector matches a pretrained one on COCO — pretraining mainly speeds convergence. In practice this needs 100K+ images and days of GPU; for a learner or a product team, always start pretrained, and consider domain pretraining when the domain is far from natural images.

Where this shows up later

• The same recipe trains detectors and segmenters on pretrained backbones — 5.5 Object detection, 5.6 Segmentation; you run it end-to-end in 5.10 Project: train and deploy an image classifier.
• Frozen embeddings + a small classifier is the pattern behind retrieval and RAG — 9.2 Embeddings deep dive; LoRA is fine-tuning with a low-rank budget — 8.2 Fine-tuning and PEFT.
• Self-supervised and contrastive pretraining supply the backbones — 5.8 Self-supervised and contrastive learning, 8.10 Multimodal models.