Derivatives, gradients and the chain rule

The derivative is the best local linear approximation

The derivative of $f$ at $x$ is defined as $f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h}$, but the way to think about it is

The derivative is the slope of the line that best matches $f$ near $x$: how much the output moves per unit nudge of the input. Training a model is nothing more than asking, for each parameter, whether a nudge lowers the loss — and the derivative is the answer. The rules you need are few:

Math check. $\sigma'(x)=\dfrac{e^{-x}}{(1+e^{-x})^2}=\dfrac{1}{1+e^{-x}}\cdot\dfrac{e^{-x}}{1+e^{-x}}=\sigma(x)\,(1-\sigma(x))$. Its maximum is $1/4$ at $x=0$ — the fact behind vanishing gradients in 4.2 Activation functions.

Partial derivatives and the gradient

A loss depends on millions of parameters, so we need derivatives of functions of many variables. The partial derivative $\partial f/\partial x_i$ nudges only $x_i$ and holds the others fixed. Collecting them gives the gradient

and the linear approximation becomes $f(\mathbf{x}+\mathbf{h})\approx f(\mathbf{x})+\nabla f(\mathbf{x})^\top\mathbf{h}$. The rate of change along a unit direction $\mathbf{u}$ — the directional derivative — is $D_{\mathbf{u}}f=\nabla f\cdot\mathbf{u}=\|\nabla f\|\cos\theta$ by 1.1 Vectors, dot products and norms. It is maximized when $\cos\theta=1$, that is, when $\mathbf{u}$ points along $\nabla f$. So the gradient is the direction of steepest ascent, its norm is the steepest slope, and $-\nabla f$ is the direction of steepest descent. Moving along a level set (where $f$ is constant) gives $D_{\mathbf{u}}f=0$, so the gradient is perpendicular to level sets. Gradient descent, $\mathbf{x}\leftarrow\mathbf{x}-\eta\,\nabla f(\mathbf{x})$, is the greedy move downhill (1.10 Optimization: from gradient descent to Adam).

Example: $f(x,y)=x^2+3y^2$ has $\nabla f=(2x,\,6y)$. At $(1,1)$ the gradient $(2,6)$ points mostly along $y$ because the bowl is steeper there; the level curves are ellipses, and the gradient crosses them at right angles.

The chain rule, on a graph

Deep networks are compositions, so the chain rule is the law of the land. In one variable, if $y=f(u)$ and $u=g(x)$ then $\dfrac{dy}{dx}=\dfrac{dy}{du}\,\dfrac{du}{dx}$: rates multiply along a chain. With several intermediate variables $u_1,\dots,u_m$ that each depend on $x$,

Multiply along each path, add over paths. Draw the computation as a graph — inputs at the bottom, loss at the top — and the rule becomes mechanical: the derivative of the output with respect to any node is the sum, over all paths to the output, of the product of edge derivatives. Computing these from the output backward and reusing shared sub-results is backpropagation (4.4 Backpropagation, fully derived); the forward pass computes values, the backward pass computes gradients, and autograd libraries do both (2.4 PyTorch I: tensors, autograd and the training loop).

Intuition. A derivative is a conversion rate: "one unit of $x$ buys $g'$ units of $u$, and one unit of $u$ buys $f'$ units of $y$", so one unit of $x$ buys $f'g'$ units of $y$. Chains multiply rates; forks add them.

Worked example: one neuron, one example

Let $z=wx+b$, $a=\sigma(z)$ and $L=(a-y)^2$ — a single sigmoid neuron trained with squared error on one example. Along the path $L\to a\to z\to w$,

Numbers: $x=2$, $y=1$, $w=0.5$, $b=0$. Forward: $z=1$, $a=\sigma(1)=0.7311$, $L=(0.7311-1)^2=0.0723$. Backward: $\partial L/\partial a=2(-0.2689)=-0.5379$; $\partial a/\partial z=0.7311\cdot0.2689=0.1966$; so $\partial L/\partial w=-0.5379\cdot0.1966\cdot2=-0.2115$ and $\partial L/\partial b=-0.1057$. A gradient step with $\eta=0.1$ moves $w$ to $0.5212$ and $b$ to $0.0106$; recomputing, $z=1.053$, $a=0.7413$, $L=0.0669$. The loss went down, as the negative gradient promised. Every parameter of every network is updated by exactly this logic, only with longer chains (4.4 Backpropagation, fully derived).

import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))
x, y, w, b = 2.0, 1.0, 0.5, 0.0
z = w * x + b; a = sig(z); L = (a - y) ** 2
dL_dw = 2 * (a - y) * a * (1 - a) * x        # -0.2115
dL_db = 2 * (a - y) * a * (1 - a)            # -0.1057

Numerical gradient checking

Derivations have bugs. The cure is to compare against a finite difference:

the central difference, whose error is $O(\varepsilon^2)$ (the one-sided version is only $O(\varepsilon)$). Use $\varepsilon\approx10^{-5}$ in double precision and compare with the relative error $\dfrac{|g_{\text{analytic}}-g_{\text{numeric}}|}{|g_{\text{analytic}}|+|g_{\text{numeric}}|}$: below $10^{-7}$ is a pass, above $10^{-4}$ is a bug. It costs two function evaluations per parameter, so it is a test, not a training method — but it is the test every autograd engine is validated against (4.4 Backpropagation, fully derived, 4.8 Training dynamics and debugging).

From gradients to learning

Put the pieces together: a model $f_\theta$, a loss $L(\theta)$ averaged over data, the gradient $\nabla_\theta L$ computed by the chain rule on the model's computation graph, and the update $\theta\leftarrow\theta-\eta\,\nabla_\theta L$. Repeat. That loop, with better step-size rules (1.10 Optimization: from gradient descent to Adam) and better architectures, is all of deep learning. The next lesson extends the calculus to vectors and matrices so that the chain rule can be written for whole layers at once.

Where this shows up later

• Jacobians, Hessians and matrix calculus — 1.5 Multivariable and matrix calculus; gradient descent and its variants — 1.10 Optimization: from gradient descent to Adam.
• Autograd in PyTorch — 2.4 PyTorch I: tensors, autograd and the training loop; backpropagation fully derived — 4.4 Backpropagation, fully derived; gradient checks as a debugging tool — 4.8 Training dynamics and debugging.
• Saturating derivatives cause vanishing gradients — 4.2 Activation functions, 6.4 Recurrent neural networks and BPTT.