Multivariable and matrix calculus

Jacobians: derivatives of vector-valued functions

For $\mathbf{f}:\mathbb{R}^n\to\mathbb{R}^m$ the Jacobian $\mathbf{J}\in\mathbb{R}^{m\times n}$ has entries $J_{ij}=\partial f_i/\partial x_j$ and gives the local linear approximation

The chain rule becomes a matrix product: $\mathbf{J}_{\mathbf{f}\circ\mathbf{g}}(\mathbf{x})=\mathbf{J}_{\mathbf{f}}(\mathbf{g}(\mathbf{x}))\,\mathbf{J}_{\mathbf{g}}(\mathbf{x})$, which is why the matrix algebra of 1.2 Matrices as transformations matters here. Three Jacobians cover most of deep learning: a linear map $\mathbf{f}(\mathbf{x})=\mathbf{A}\mathbf{x}$ has $\mathbf{J}=\mathbf{A}$; an element-wise function $f_i=\phi(x_i)$ has $\mathbf{J}=\mathrm{diag}(\phi'(x_1),\dots,\phi'(x_n))$; and a layer $\phi(\mathbf{W}\mathbf{x})$ therefore has $\mathbf{J}=\mathrm{diag}(\phi'(\mathbf{W}\mathbf{x}))\,\mathbf{W}$. A scalar function is the case $m=1$: its Jacobian is the row vector $\nabla f^\top$.

Hessians, Taylor expansion and curvature

Going one order further gives the quadratic model of a function near $\mathbf{x}$:

The Hessian $\mathbf{H}$ is symmetric (mixed partials commute), so by the spectral theorem of 1.3 Eigen-things, SVD and low rank its eigenvalues are real and its eigenvectors orthogonal: each eigenvalue is the curvature of $f$ along its eigenvector. At a critical point ($\nabla f=\mathbf{0}$), $\mathbf{H}\succ0$ means a local minimum, $\mathbf{H}\prec0$ a maximum, and mixed signs a saddle — for $f(x,y)=x^2-y^2$, $\mathbf{H}=\mathrm{diag}(2,-2)$ and the origin is a saddle. A function is convex exactly when $\mathbf{H}\succeq0$ everywhere; then every local minimum is global, which is why the classical models of Phase 3 are so well behaved. Minimizing the quadratic model gives Newton's step $\boldsymbol{\delta}=-\mathbf{H}^{-1}\nabla f$, which solves a quadratic in one move. With $d$ parameters the Hessian has $d^2$ entries, impossible at $d=10^9$, so deep learning uses first-order methods with cheap curvature estimates such as Adam's second moments (1.10 Optimization: from gradient descent to Adam, 4.5 Optimizers and learning-rate schedules).

Intuition. The gradient tells you which way is down; the Hessian tells you how the slope changes — how far you can safely step before the ground curves back up.

Layout conventions: read this before any paper

There are two conventions for $\partial\mathbf{y}/\partial\mathbf{x}$: numerator layout (the $m\times n$ Jacobian above) and denominator layout (its transpose). Papers mix them, which is the source of most "my shapes are transposed" confusion. The convention every ML library uses in code is simpler and worth memorizing: the gradient of a scalar loss with respect to any tensor has the same shape as that tensor, so you can subtract it in a gradient step. This lesson writes Jacobians in numerator layout and gradients as same-shape tensors.

The identities you actually need

The last two rows are backpropagation through a linear layer; you will use them in 4.4 Backpropagation, fully derived without thinking. A reliable way to recover them is the shape trick: $\partial L/\partial\mathbf{W}$ must have the shape of $\mathbf{W}$ ($d_{\text{out}}\times d_{\text{in}}$), and the only product of $\mathbf{g}\in\mathbb{R}^{d_{\text{out}}}$ and $\mathbf{x}\in\mathbb{R}^{d_{\text{in}}}$ with that shape is the outer product $\mathbf{g}\mathbf{x}^\top$.

Derivation 1: the least-squares gradient

Expand $L(\mathbf{w})=\|\mathbf{X}\mathbf{w}-\mathbf{y}\|^2=(\mathbf{X}\mathbf{w}-\mathbf{y})^\top(\mathbf{X}\mathbf{w}-\mathbf{y})=\mathbf{w}^\top\mathbf{X}^\top\mathbf{X}\mathbf{w}-2\,\mathbf{y}^\top\mathbf{X}\mathbf{w}+\mathbf{y}^\top\mathbf{y}$. The first term is a quadratic form with the symmetric matrix $\mathbf{X}^\top\mathbf{X}$ (gradient $2\mathbf{X}^\top\mathbf{X}\mathbf{w}$), the second is linear in $\mathbf{w}$ with $\mathbf{a}=2\mathbf{X}^\top\mathbf{y}$, the third is constant:

Setting it to zero gives the normal equations $\mathbf{X}^\top\mathbf{X}\,\mathbf{w}=\mathbf{X}^\top\mathbf{y}$ (3.2 Linear regression). The Hessian is $2\mathbf{X}^\top\mathbf{X}\succeq0$, so the problem is convex and this critical point is the global minimum. Notice the pattern "residual pushed back through $\mathbf{X}^\top$" — it is the batch backward rule from the table.

Derivation 2: softmax with cross-entropy

For logits $\mathbf{z}\in\mathbb{R}^K$, the softmax is $p_i=\dfrac{e^{z_i}}{\sum_j e^{z_j}}$ and the cross-entropy loss against a one-hot target $\mathbf{y}$ is $L=-\sum_i y_i\log p_i$. The softmax Jacobian is

where $\delta_{ij}$ is 1 when $i=j$ and 0 otherwise. Then by the chain rule,

So $\nabla_{\mathbf{z}}L=\mathbf{p}-\mathbf{y}$: prediction minus target. The same form appears in logistic regression (3.3 Logistic regression and softmax classification), and it is why frameworks fuse softmax and cross-entropy into one numerically stable operation (4.3 Loss functions catalog).

Math check. For $i=j$: $\partial p_i/\partial z_i=\dfrac{e^{z_i}\sum_j e^{z_j}-e^{z_i}e^{z_i}}{(\sum_j e^{z_j})^2}=p_i-p_i^2=p_i(1-p_i)$. For $i\neq j$: $-\dfrac{e^{z_i}e^{z_j}}{(\sum_k e^{z_k})^2}=-p_ip_j$. Both match $p_i(\delta_{ij}-p_j)$. ✓

Numbers: $\mathbf{z}=(2,1,0)$, $\mathbf{y}=(1,0,0)$. Then $e^{\mathbf{z}}=(7.389,\,2.718,\,1)$ with sum $11.107$, so $\mathbf{p}=(0.665,\,0.245,\,0.090)$, $L=-\ln0.665=0.408$ and $\nabla_{\mathbf{z}}L=\mathbf{p}-\mathbf{y}=(-0.335,\,0.245,\,0.090)$: the gradient pushes the correct logit up and the others down, in proportion to how much probability they wrongly took.

Vector-Jacobian products: what autograd really computes

For a chain $L=\ell\big(\mathbf{f}_L(\cdots\mathbf{f}_1(\mathbf{x}))\big)$ the gradient is

Reverse-mode differentiation evaluates this from the right: starting with the vector $\nabla\ell$, each step is a vector-Jacobian product $\mathbf{v}^\top\mathbf{J}_k$ — a matrix-vector operation that never forms $\mathbf{J}_k$ itself (which for a layer with a million inputs and outputs would have $10^{12}$ entries). The whole backward pass costs a small constant times the forward pass, independent of how many parameters you have, which is the economic miracle that makes deep learning possible. Its price is memory: every layer's activations must be kept until the backward pass reaches them (4.4 Backpropagation, fully derived, 4.12 Compute, hardware and scale basics). Forward mode computes Jacobian-vector products $\mathbf{J}\mathbf{v}$ instead, cheap when inputs are few and outputs many — the opposite of a loss function, which is why ML libraries default to reverse mode.

Where this shows up later

• Normal equations and ridge regression — 3.2 Linear regression, 3.6 Regularization and generalization; logistic-regression gradient $\mathbf{X}^\top(\mathbf{p}-\mathbf{y})$ — 3.3 Logistic regression and softmax classification; loss functions on logits — 4.3 Loss functions catalog.
• Backpropagation through linear layers uses exactly the table above — 4.4 Backpropagation, fully derived; autograd in PyTorch — 2.4 PyTorch I: tensors, autograd and the training loop.
• Curvature, conditioning and Adam's curvature estimates — 1.10 Optimization: from gradient descent to Adam, 4.5 Optimizers and learning-rate schedules; memory cost of stored activations — 4.12 Compute, hardware and scale basics.