Matrices as transformations

A matrix is a function

A matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$ has $m$ rows and $n$ columns and defines a function $\mathbf{x}\mapsto\mathbf{A}\mathbf{x}$ from $\mathbb{R}^n$ to $\mathbb{R}^m$. There are two ways to compute the product, and you need both:

The column picture says: the $j$-th column is where the basis vector $\mathbf{e}_j$ lands, and every other input lands on the corresponding combination. The row picture says: each output coordinate is a dot product of the input with one row — a template match (1.1 Vectors, dot products and norms). The function is linear, $\mathbf{A}(c\mathbf{x}+d\mathbf{y})=c\mathbf{A}\mathbf{x}+d\mathbf{A}\mathbf{y}$: lines stay lines, the origin stays put, grid lines stay parallel and evenly spaced. That is exactly what the demo above draws when you move the sliders.

Example: $\mathbf{A}=\begin{bmatrix}2&1\\0&1\end{bmatrix}$ sends $\mathbf{e}_1=(1,0)$ to $(2,0)$ and $\mathbf{e}_2=(0,1)$ to $(1,1)$ — a horizontal stretch combined with a shear.

Intuition. To know what a matrix does, you only need to know where it sends the basis vectors. The columns are the transformation.

Composition is multiplication

Apply $\mathbf{B}$ and then $\mathbf{A}$: $\mathbf{A}(\mathbf{B}\mathbf{x})=(\mathbf{A}\mathbf{B})\mathbf{x}$. The product's entries are

defined only when the inner dimensions agree: $(m\times n)(n\times p)=m\times p$. Read $\mathbf{A}\mathbf{B}$ right to left — $\mathbf{B}$ acts first. Multiplication is associative, $(\mathbf{A}\mathbf{B})\mathbf{C}=\mathbf{A}(\mathbf{B}\mathbf{C})$, but not commutative: rotating then stretching is not stretching then rotating. With $\mathbf{A}$ above and the 90° rotation $\mathbf{B}=\begin{bmatrix}0&-1\\1&0\end{bmatrix}$,

A deep network is a long composition $f_L\circ\dots\circ f_1$; between the nonlinearities it is exactly matrix products, which is why shapes must chain and why gradients multiply along the chain (4.4 Backpropagation, fully derived, 6.4 Recurrent neural networks and BPTT).

Transpose and the special matrices

The transpose flips rows and columns, $(\mathbf{A}^\top)_{ij}=A_{ji}$, and reverses products: $(\mathbf{A}\mathbf{B})^\top=\mathbf{B}^\top\mathbf{A}^\top$. Two products of vectors look alike and are opposites: $\mathbf{x}^\top\mathbf{y}$ is the dot product (a scalar) while $\mathbf{x}\mathbf{y}^\top$ is the outer product, a rank-one matrix whose every column is a multiple of $\mathbf{x}$. The matrices with names are the ones whose geometry you should recognize on sight:

Inverse, rank, column space and null space

The column space $\mathcal{C}(\mathbf{A})$ is the span of the columns — the set of outputs the transformation can reach. Its dimension is the rank. The null space $\mathcal{N}(\mathbf{A})=\{\mathbf{x}:\mathbf{A}\mathbf{x}=\mathbf{0}\}$ is everything squashed to the origin. They are tied by rank–nullity: $\operatorname{rank}(\mathbf{A})+\dim\mathcal{N}(\mathbf{A})=n$. A square matrix is invertible when there is an $\mathbf{A}^{-1}$ with $\mathbf{A}^{-1}\mathbf{A}=\mathbf{A}\mathbf{A}^{-1}=\mathbf{I}$, which happens exactly when the rank is full: nothing is squashed and every output is reachable, so the transformation can be undone. Solving $\mathbf{A}\mathbf{x}=\mathbf{b}$ then has one solution; if $\mathbf{b}\notin\mathcal{C}(\mathbf{A})$ there is none, and if the null space is nontrivial and $\mathbf{b}$ is reachable there are infinitely many. For $2\times2$ matrices,

In practice you almost never form an inverse explicitly: you solve the system (np.linalg.solve), which is faster and more numerically stable. Data matrices in ML are tall and rectangular, so "inverse" becomes "least squares" (3.2 Linear regression).

The determinant: how much area scales

For a square matrix, $\det\mathbf{A}$ is the signed factor by which areas (volumes in higher dimensions) scale: the unit square becomes a parallelogram of area $|\det\mathbf{A}|$, and a negative sign means orientation flipped. For $2\times2$, $\det=ad-bc$. The rules follow from the geometry: $\det(\mathbf{A}\mathbf{B})=\det\mathbf{A}\,\det\mathbf{B}$ (areas multiply under composition), $\det\mathbf{A}^\top=\det\mathbf{A}$, and $\det\mathbf{A}^{-1}=1/\det\mathbf{A}$. The crucial fact: $\det\mathbf{A}=0$ exactly when $\mathbf{A}$ squashes space into a lower dimension, which is exactly when it is not invertible. For the matrix above, $\det=2\cdot1-1\cdot0=2$: areas double. Determinants of covariance matrices normalize the Gaussian density (1.7 Probability II: Bayes, independence, MLE and MAP), and the eigenvalues of 1.3 Eigen-things, SVD and low rank multiply to the determinant.

Data as a matrix, and the linear layer

Stack $n$ examples with $d$ features as rows: $\mathbf{X}\in\mathbb{R}^{n\times d}$. A linear layer with $k$ outputs has weights $\mathbf{W}\in\mathbb{R}^{k\times d}$ and bias $\mathbf{b}\in\mathbb{R}^k$ (PyTorch's convention) and computes, for the whole batch at once,

where the bias is added to every row (broadcasting, 2.2 NumPy and vectorized thinking). Each output column is a template match of every example against one row of $\mathbf{W}$: $k$ neurons evaluated on $n$ inputs in a single product. Shapes: $(n\times d)(d\times k)=(n\times k)$. Parameters: $kd+k$. Cost: about $2ndk$ floating-point operations (one multiply and one add per term). For a batch of 32 MNIST images ($d=784$) and $k=128$ hidden units, $\mathbf{Z}$ is $32\times128$, the layer has $784\cdot128+128=100{,}480$ parameters and costs $2\cdot32\cdot784\cdot128\approx6.4$ million flops. A transformer is mostly this operation (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings); attention scores are the same product between queries and keys, $\mathbf{Q}\mathbf{K}^\top$ (7.2 Scaled dot-product and multi-head attention, fully derived).

Worked example, end to end

Take $\mathbf{A}=\begin{bmatrix}2&1\\0&1\end{bmatrix}$ and $\mathbf{x}=(1,1)$.

• Column picture: $\mathbf{A}\mathbf{x}=1\cdot(2,0)+1\cdot(1,1)=(3,1)$. Row picture: $(2\cdot1+1\cdot1,\;0\cdot1+1\cdot1)=(3,1)$. ✓
• $\det\mathbf{A}=2$ and $\mathbf{A}^{-1}=\tfrac12\begin{bmatrix}1&-1\\0&2\end{bmatrix}$. Check: $\mathbf{A}^{-1}(3,1)=\tfrac12(3-1,\;2)=(1,1)$. ✓
• Rank 2, null space $\{\mathbf{0}\}$, column space all of $\mathbb{R}^2$.
• $\mathbf{A}^\top\mathbf{A}=\begin{bmatrix}4&2\\2&2\end{bmatrix}$ is symmetric and positive definite, because $\mathbf{x}^\top\mathbf{A}^\top\mathbf{A}\mathbf{x}=\|\mathbf{A}\mathbf{x}\|^2>0$ for $\mathbf{x}\neq\mathbf{0}$ — the pattern behind the normal equations (3.2 Linear regression) and the SVD (1.3 Eigen-things, SVD and low rank).

Math check. $\det(\mathbf{A}^\top\mathbf{A})=4\cdot2-2\cdot2=4=(\det\mathbf{A})^2$. ✓

Where this shows up later

• NumPy broadcasting, @ and einsum — 2.2 NumPy and vectorized thinking; PyTorch linear layers — 2.4 PyTorch I: tensors, autograd and the training loop, 4.1 From perceptron to MLP; universal approximation.
• Normal equations and projections — 3.2 Linear regression; eigenvectors, SVD and low rank — 1.3 Eigen-things, SVD and low rank; PCA — 3.12 Dimensionality reduction: PCA, t-SNE and UMAP.
• Attention is a matrix product of queries, keys and values — 7.2 Scaled dot-product and multi-head attention, fully derived; parameter counting — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; products of Jacobians cause vanishing gradients — 6.4 Recurrent neural networks and BPTT.