4.2 Activation functions

The activation function is a one-line decision with network-wide consequences: it decides whether a 50-layer network trains at all. Sigmoids made deep networks untrainable for twenty years; ReLU unlocked AlexNet (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt); GELU and SwiGLU sit inside every modern transformer (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings). Knowing each function's derivative and saturation behaviour is what lets you read a stalled training run (4.8 Training dynamics and debugging) and pick the right default without guessing.

You'll be able to

Write the formula and derivative of sigmoid, tanh, ReLU and its variants, GELU/SiLU and softmax
Derive why products of layer Jacobians make gradients vanish with saturating activations and why ReLU avoids it
Explain dead ReLUs, their cause and their cure
Match output activations to tasks and losses, and explain why models should output logits
State the 2026 defaults for hidden activations in CNNs, MLPs and transformers

Key ideas

Sigmoid and tanh saturate: their derivatives are at most 1/4 and 1, and near zero for large |z|.
Backprop multiplies one Jacobian diag(φ'(z)) per layer; with saturating activations the product shrinks geometrically — the vanishing-gradient problem.
ReLU's derivative is exactly 1 on active units, so gradients pass through unattenuated; this unlocked deep networks in 2012.

Show 4 more key ideas

A ReLU unit whose pre-activation is negative for every input is dead forever; large learning rates and bad initialization cause it; leaky/GELU variants avoid it.
GELU = z·Φ(z) and SiLU = z·σ(z) are smooth, slightly non-monotonic ReLUs; GELU/SwiGLU are the transformer defaults.
Softmax turns logits into probabilities with Jacobian ∂pᵢ/∂zⱼ = pᵢ(δᵢⱼ − pⱼ); it is shift-invariant and temperature-scalable.
Models should output logits; the loss function applies sigmoid/softmax stably.

Suggested path through this lesson

What we need from a nonlinearity

An activation $\phi$ is applied element-wise to the pre-activations $\mathbf{z}$ of a layer (4.1 From perceptron to MLP; universal approximation). It must be nonlinear (or the network collapses to a linear model), cheap, and — the requirement that took the field decades to appreciate — it must let gradients pass through many layers. Every property below is about that third requirement.

The catalog, with derivatives

activation	formula	derivative	notes
sigmoid	$\sigma(z)=\dfrac{1}{1+e^{-z}}$	$\sigma(z)(1-\sigma(z))$ , at most $\tfrac14$	outputs in $(0,1)$ ; saturates for $\lvert z\rvert \gtrsim 4$
tanh	$\tanh(z)=2\sigma(2z)-1$	$1-\tanh^2(z)$ , at most $1$	zero-centred version of sigmoid; still saturates
ReLU	$\max(0,z)$	$1$ if $z>0$ else $0$	no saturation for $z>0$ ; "dead" when $z<0$
Leaky ReLU	$\max(\alpha z, z)$ , $\alpha\approx0.01$	$1$ or $\alpha$	small negative slope keeps units alive
PReLU	Leaky ReLU with learnable $\alpha$	$1$ or $\alpha$	one extra parameter per channel
ELU	$z$ if $z>0$ else $\alpha(e^{z}-1)$	$1$ or $\alpha e^{z}$	smooth, mean closer to zero
Softplus	$\log(1+e^{z})$	$\sigma(z)$	smooth ReLU; used for positive outputs
SiLU / Swish	$z\,\sigma(z)$	$\sigma(z)\big(1+z(1-\sigma(z))\big)$	smooth, non-monotonic; Llama-family MLPs
GELU	$z\,\Phi(z)$ , $\Phi$ = standard normal CDF	$\Phi(z)+z\,\varphi(z)$	BERT, GPT-2/3, ViT default

GELU is usually computed with the tanh approximation $0.5\,z\,\big(1+\tanh[\sqrt{2/\pi}\,(z+0.044715\,z^{3})]\big)$ . It can be read as the expected value of a stochastic gate: multiply $z$ by $\mathbb{1}[X\le z]$ with $X\sim\mathcal{N}(0,1)$ , so inputs are kept with probability $\Phi(z)$ — larger inputs are kept more often, which is why GELU is a smooth, "soft" ReLU that dips slightly negative around $z\approx-0.75$ .

Softmax is the one activation that is not element-wise. It turns a vector of logits into a probability distribution,

p_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}, \qquad \frac{\partial p_i}{\partial z_j} = p_i\,(\delta_{ij}-p_j),

where $\delta_{ij}$ is 1 if $i=j$ and 0 otherwise. It is shift-invariant ( $\mathrm{softmax}(\mathbf{z}+c)=\mathrm{softmax}(\mathbf{z})$ ), which is what makes the log-sum-exp trick of 4.3 Loss functions catalog possible, and dividing the logits by a temperature $T$ sharpens or flattens it (8.5 Decoding and sampling).

Gated linear units multiply two linear projections, one of them squashed: $\mathrm{GLU}(\mathbf{x}) = (\mathbf{W}\mathbf{x})\odot\sigma(\mathbf{V}\mathbf{x})$ . Replace $\sigma$ with SiLU and you get SwiGLU, the feed-forward nonlinearity in Llama, PaLM and most 2024–26 open models (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings). The gate lets one projection decide how much of the other to let through.

Saturation and the vanishing-gradient argument

Here is the calculation that explains twenty years of history. In an $L$ -layer network, backpropagation (4.4 Backpropagation, fully derived) multiplies one Jacobian per layer. Writing $\mathbf{D}^{(\ell)} = \mathrm{diag}\big(\phi'(\mathbf{z}^{(\ell)})\big)$ , the gradient reaching layer 1 is

\frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(1)}} = \mathbf{W}^{(2)\top}\mathbf{D}^{(2)}\;\mathbf{W}^{(3)\top}\mathbf{D}^{(3)}\cdots\mathbf{W}^{(L)\top}\mathbf{D}^{(L)}\;\frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(L)}} .

Each factor $\mathbf{D}^{(\ell)}$ scales the signal by the derivatives of the activation. For sigmoid, $\phi'\le\tfrac14$ everywhere and is far smaller once units saturate. If the weight matrices have roughly unit scale, the gradient norm shrinks by at least a factor of 4 per layer: after 10 layers it is at most $4^{-10}\approx 10^{-6}$ of the output gradient, and the early layers effectively stop learning. That is the vanishing gradient problem. If instead the weights are large, the product grows geometrically — exploding gradients, which show up as loss spikes and NaNs (4.8 Training dynamics and debugging) and are tamed by clipping (4.5 Optimizers and learning-rate schedules).

Math check. Put numbers in at $z=2$ : $\sigma'(2)=0.105$ , $\tanh'(2)=0.071$ , $\mathrm{ReLU}'(2)=1$ , $\mathrm{SiLU}'(2)\approx1.09$ , $\mathrm{GELU}'(2)\approx1.09$ . A sigmoid unit that is only moderately active already passes on a tenth of the gradient; a ReLU passes all of it.

ReLU's derivative is exactly 1 on every active unit, so the $\mathbf{D}$ factors become masks rather than attenuators: the gradient reaching an early layer is a sum over "active paths" with no shrinkage along them. That single property is why AlexNet (2012) could train 8 layers with plain SGD when sigmoid networks of that depth could not (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt). Going much deeper still required initialization that keeps the weight factors near unit scale (4.6 Initialization and normalization) and residual connections (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt).

Dead ReLUs

The price of a zero derivative for $z<0$ : if a unit's pre-activation is negative for every input in the dataset, its gradient is identically zero and no update can ever revive it — the unit is dead. Deaths happen when a large learning rate (or a large gradient) knocks a bias far negative, or when initialization is too aggressive. Symptoms: a growing fraction of units with zero activation on every batch (4.8 Training dynamics and debugging). Cures: lower the learning rate, use a leaky variant, or use GELU/SiLU whose derivatives are never exactly zero. In practice a few percent of dead units is normal; tens of percent is a bug.

Output activations: match the task, output logits

task	output activation	loss (4.3 Loss functions catalog)
regression	identity	MSE / Huber
binary classification	sigmoid	binary cross-entropy
multi-class (one label)	softmax	categorical cross-entropy
multi-label	sigmoid per label	BCE per label
positive quantity	softplus / exp	Gaussian / Poisson NLL

The model itself should return logits; PyTorch's nn.CrossEntropyLoss and nn.BCEWithLogitsLoss apply the sigmoid/softmax internally in a numerically stable way. Applying the activation in the model and then again in the loss is one of the most common silent bugs in deep learning: the network still trains, just much worse.

Choosing an activation in 2026

Hidden layers of CNNs and MLPs: ReLU is the default; swap to Leaky ReLU or GELU if you see dead units.
Transformers: GELU (BERT/GPT-2 lineage) or SwiGLU (Llama lineage) — both are smooth, which seems to matter for very large models.
Gates and recurrent cells: sigmoid and tanh survive exactly where a bounded output is needed — LSTM/GRU gates (6.5 LSTM and GRU), attention softmax (7.2 Scaled dot-product and multi-head attention, fully derived), mixture-of-experts routers (7.11 Mixture of Experts).
Outputs: by the table above, implemented inside the loss.

Activations are not the place to spend tuning budget; initialization, normalization and the learning-rate schedule (4.5 Optimizers and learning-rate schedules, 4.6 Initialization and normalization) matter far more once you are using a modern default.

Where this shows up later

The log-sum-exp stabilization of softmax + cross-entropy — 4.3 Loss functions catalog; the Jacobian product made precise in the backward recursion — 4.4 Backpropagation, fully derived.
Keeping pre-activations in the non-saturating range by design — initialization and normalization in 4.6 Initialization and normalization; diagnosing saturation and dead units — 4.8 Training dynamics and debugging.
ReLU in AlexNet/ResNet — 5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt; GELU and SwiGLU in the transformer feed-forward block — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; softmax with temperature in decoding — 8.5 Decoding and sampling.

Common pitfalls (4)

Applying softmax inside the model and again inside nn.CrossEntropyLoss — training 'works' but is badly degraded.
Using sigmoid hidden units in a deep network and then wondering why early layers don't learn.
Treating ReLU's zero derivative as harmless: a high learning rate can kill a large fraction of units permanently.
Assuming the choice of hidden activation is the main hyperparameter — initialization, normalization and learning rate dominate.

Recap: all 7 key points

Sigmoid and tanh saturate: their derivatives are at most 1/4 and 1, and near zero for large |z|.
Backprop multiplies one Jacobian diag(φ'(z)) per layer; with saturating activations the product shrinks geometrically — the vanishing-gradient problem.
ReLU's derivative is exactly 1 on active units, so gradients pass through unattenuated; this unlocked deep networks in 2012.
A ReLU unit whose pre-activation is negative for every input is dead forever; large learning rates and bad initialization cause it; leaky/GELU variants avoid it.
GELU = z·Φ(z) and SiLU = z·σ(z) are smooth, slightly non-monotonic ReLUs; GELU/SwiGLU are the transformer defaults.
Softmax turns logits into probabilities with Jacobian ∂pᵢ/∂zⱼ = pᵢ(δᵢⱼ − pⱼ); it is shift-invariant and temperature-scalable.
Models should output logits; the loss function applies sigmoid/softmax stably.

Watch

Neural Networks Pt. 3: ReLU In Action

StatQuest · 9 min · open on YouTube ↗

How ReLU units bend and add up to a nonlinear fit.

4 videos

Read

Courses

CS231n notes — Neural Networks Part 1: commonly used activation functions ↗Stanford CS231n · 2024Compact pros/cons of sigmoid, tanh, ReLU, Leaky ReLU and Maxout.

Papers

Gaussian Error Linear Units (GELUs) ↗Hendrycks & Gimpel · 2016The stochastic-gate derivation of GELU and the tanh approximation.GLU Variants Improve Transformer ↗Noam Shazeer · 2020Two pages that introduced SwiGLU/GEGLU; note the famous non-explanation in the conclusion.Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification ↗He, Zhang, Ren & Sun · 2015PReLU and the initialization derived for ReLU networks (previewed in [[4.6]]).

Books

Understanding Deep Learning — Section 3.5 and Chapter 7 notes on activations ↗Simon J.D. Prince · 2023Figures comparing activation families and their effect on the piecewise structure.

Try it live

Interactive demo

runs in your browser

Loading the demo…

Exercises

By hand: derivatives and saturation
Derive $\sigma'(z)=\sigma(z)(1-\sigma(z))$ and $\tanh'(z)=1-\tanh^2(z)$ from the definitions. Then evaluate each of sigmoid, tanh, ReLU and SiLU and their derivatives at $z\in\{-4,-1,0,1,4\}$ . Expected: sigmoid' peaks at 0.25 at $z=0$ and is below 0.02 at $\lvert z\rvert=4$ .
NumPy: vanishing gradients in a 20-layer stack
Build a 20-layer network of width 100 with Gaussian weights of standard deviation $1/\sqrt{100}$ and random inputs. Compute the gradient of the mean output with respect to the input (or just the product of Jacobians) with sigmoid, tanh and ReLU activations. Plot the gradient norm per layer. Expected: sigmoid shrinks by roughly 4× per layer; ReLU stays within an order of magnitude.
PyTorch: count dead ReLUs
Train a 784-512-512-10 MLP on MNIST for one epoch at learning rates 0.01, 0.1 and 1.0 (SGD). After training, pass the test set through and compute, per hidden unit, the fraction of inputs for which the activation is 0; count units dead on 100% of inputs. Expected: the fraction of dead units rises sharply with learning rate.

Quiz

Question 1 of 5

What is the maximum value of the derivative of the sigmoid function?

Flashcards

Card 1 of 8

These cards are now in your spaced-repetition queue. Review all due cards →

Scored 80% or more and reviewed the cards? You're done here.