Interpretability, robustness and adversarial examples

Why look inside

Accuracy on a held-out split only tells you the model works on data like the training data. It does not tell you why it works. Classic failures: a wolf-vs-husky classifier that detects snow; a pneumonia model that reads the portable-X-ray marker; a "tench" detector that fires on the fisherman's fingers. Interpretability tools come in two families — attribution (which input pixels mattered for this prediction?) and feature visualization (what pattern does this neuron or channel detect?) — plus behavioral tests (adversarial and shifted inputs).

Attribution by gradients

The simplest attribution is the gradient saliency map (Simonyan et al., 2013): $S = \left|\partial y_c / \partial \mathbf{x}\right|$ (max over color channels), a first-order answer to "which pixels would change the class score most?". It is noisy because ReLU networks are piecewise linear and gradients flicker; SmoothGrad averages gradients over noisy copies of the input. Integrated Gradients (Sundararajan et al., 2017) fixes a deeper flaw — gradients saturate — by integrating along a straight path from a baseline $\mathbf{x}'$ (a black image) to the input:

with the completeness property $\sum_i \mathrm{IG}_i = F(\mathbf{x}) - F(\mathbf{x}')$: attributions add up to the change in output, so they are on an interpretable scale.

Grad-CAM, derived

Pixel-level maps are noisy; Grad-CAM (2016) works at the level of the last convolutional feature maps $A^k \in \mathbb{R}^{u\times v}$, which are spatial and semantic. For class $c$ with pre-softmax score $y^c$:

1. Compute the gradient of the score with respect to each feature map and global-average-pool it to get a channel importance:

2. Combine the feature maps with these weights and keep only positive evidence:

3. Upsample the $u\times v$ map to the image size and overlay it.

Why does this work? For a network ending in GAP + linear layer, $y^c = \sum_k w^c_k \cdot \frac{1}{Z}\sum_{ij}A^k_{ij}$, so $\partial y^c/\partial A^k_{ij} = w^c_k / Z$ and $\alpha^c_k = w^c_k$ exactly: Grad-CAM reduces to the original CAM, and generalizes it to any architecture (with FC layers, for detection, for captioning) via gradients. The ReLU discards regions that argue against the class. Combining with guided backprop gives sharper "Guided Grad-CAM". A $5$-line call to pytorch-grad-cam does all this; use it on your 5.10 Project: train and deploy an image classifier model.

Feature visualization

Instead of asking which pixels matter for one image, synthesize an input that maximally activates a neuron, channel or logit: start from noise, run gradient ascent on the activation, and regularize (random jitter/rotation/scaling, parametrizing the image in a decorrelated Fourier basis) to avoid adversarial high-frequency noise. The Distill "Feature Visualization" article shows the resulting progression — edges → textures → parts → objects — and dataset examples (top-activating patches) complement it. Network Dissection quantifies how many channels align with human concepts. These ideas later became the circuits program in LLMs (8.11 Interpretability: what is happening inside).

Adversarial examples

Add an imperceptible perturbation and the prediction flips. The Fast Gradient Sign Method (Goodfellow et al., 2014):

one gradient step that moves every pixel by $\pm\epsilon$ in the direction that increases the loss. With $\epsilon = 0.007$ (about 2 of 255 gray levels), GoogLeNet's "panda" became "gibbon" with 99.3% confidence. The explanation is linearity in high dimensions: a linear score $\mathbf{w}^\top\mathbf{x}$ changes by $\mathbf{w}^\top\boldsymbol\eta$, and choosing $\boldsymbol\eta = \epsilon\,\mathrm{sign}(\mathbf{w})$ changes it by $\epsilon\lVert\mathbf{w}\rVert_1$ — a quantity that grows with dimension even though each coordinate moved by a tiny $\epsilon$ (1.1 Vectors, dot products and norms). PGD iterates small FGSM steps and projects back onto the $\epsilon$-ball; it is the standard strong attack. Adversarial examples transfer between models (enabling black-box attacks), survive printing and camera capture (adversarial patches, stickers on stop signs), and exist for every modality. The best known defense is adversarial training — solving the min–max problem

with PGD generating the inner maximization — at the cost of clean accuracy and training time. Randomized smoothing gives certified robustness radii. Many proposed defenses only masked gradients and were broken within months; evaluate against adaptive attacks. The LLM analog is prompt injection: adversarial text that flips an agent's behavior (8.12 Safety, risks and responsible AI, 10.10 Agent safety and security).

Distribution shift and shortcut learning

Models exploit whatever correlates with the label in training data: texture (ImageNet CNNs classify a cat-shaped image with elephant texture as an elephant), backgrounds (cows on beaches are not recognized), hospital-specific artifacts. Geirhos et al. call this shortcut learning. Measure it with shifted test sets: ImageNet-C (corruptions), ImageNet-R (renditions), ImageNet-A (natural adversarial examples), ImageNet-Sketch, and in your own domain by holding out an entire site/device/season. Mitigations: more diverse data, augmentation that breaks the shortcut (style transfer for texture bias), group-balanced training, pretraining at scale (CLIP and DINOv2 are markedly more robust), and reporting worst-group rather than average accuracy.

Calibration and uncertainty

A calibrated model's confidence matches its accuracy: of the predictions made with 80% confidence, 80% should be right. Modern deep nets are over-confident (Guo et al., 2017). Measure with a reliability diagram and the Expected Calibration Error; fix cheaply with temperature scaling — fit a single scalar $T$ on the validation set and predict with $\mathrm{softmax}(\mathbf{z}/T)$ (8.5 Decoding and sampling uses the same knob for sampling). For genuine uncertainty estimates use deep ensembles or MC dropout; for out-of-distribution detection, energy scores or feature-space distances. Calibration is what lets you set a threshold for "ask a human" (3.5 Evaluating models: metrics, validation and the bias–variance trade-off).

Data and fairness audits

ImageNet's labels are about 6% wrong; face datasets have skewed demographics that translate into unequal error rates. Before trusting a model, audit the data: class balance, duplicates between train and test, label noise (confident-learning tools), demographic and geographic coverage, and document it (datasheets, 3.13 Feature engineering, data quality and data-centric ML). Then audit the model by subgroup, not just on average.

Where this shows up later

• Mechanistic interpretability of transformers (circuits, superposition, sparse autoencoders) — 8.11 Interpretability: what is happening inside.
• Adversarial inputs for LLMs: jailbreaks and prompt injection — 8.12 Safety, risks and responsible AI, 10.10 Agent safety and security.
• Evaluation and calibration fundamentals — 3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 8.7 Evaluating LLMs; temperature in decoding — 8.5 Decoding and sampling.