From perceptron to MLP; universal approximation

A neuron is a thresholded dot product

In 1958 Frank Rosenblatt's perceptron took an input vector $\mathbf{x}\in\mathbb{R}^d$, formed the dot product with a weight vector, added a bias, and fired if the result was positive:

You already know what this computes (1.1 Vectors, dot products and norms): $\mathbf{w}$ is a template, and $\mathbf{w}^\top\mathbf{x}$ scores how well the input matches it. Geometrically, $\mathbf{w}$ is the normal vector of a hyperplane $\mathbf{w}^\top\mathbf{x}+b=0$ that splits the input space in two. The perceptron learning rule nudges the template toward misclassified positives and away from misclassified negatives: for each example with label $y\in\{0,1\}$,

The perceptron convergence theorem guarantees that if the data are linearly separable, this rule makes finitely many mistakes and stops. Logistic regression (3.3 Logistic regression and softmax classification) is the same model with the hard threshold replaced by a sigmoid, trained by gradient descent; the perceptron is where the word neuron and the idea of learning weights from errors enter the story.

XOR: the wall a single layer hits

Consider the exclusive-or function on two binary inputs:

The two positive points sit on one diagonal and the two negatives on the other: no single line separates them, so no perceptron (and no logistic regression on raw inputs) can represent XOR. Minsky and Papert's 1969 book made this precise and helped freeze neural-network research for a decade. The fix is a hidden layer. Take two hidden units with ReLU activations $\mathrm{ReLU}(z)=\max(0,z)$:

Look at what happened in the hidden space: the two positive inputs both map to $(h_1,h_2)=(1,0)$, while the negatives map to $(0,0)$ and $(2,1)$. In those new coordinates the classes are linearly separable, and the output unit is an ordinary perceptron. The hidden layer did not "solve" XOR by being cleverer; it re-represented the data so that a linear rule works. That is representation learning in miniature, and it is the entire idea behind deep learning.

The multilayer perceptron, written out

An MLP with $L$ layers applies an affine map followed by an element-wise nonlinearity $\phi$, $L$ times. With $\mathbf{h}^{(0)}=\mathbf{x}$:

where $\mathbf{W}^{(\ell)}\in\mathbb{R}^{n_\ell\times n_{\ell-1}}$ and $\mathbf{b}^{(\ell)}\in\mathbb{R}^{n_\ell}$. The last layer usually skips $\phi$ and feeds $\mathbf{z}^{(L)}$ (the logits) into a softmax for classification or uses it directly for regression. $\mathbf{z}$ is called the pre-activation and $\mathbf{h}$ the activation; you will differentiate with respect to both in 4.4 Backpropagation, fully derived.

For a mini-batch of $B$ examples stacked as rows of $\mathbf{X}\in\mathbb{R}^{B\times d}$ (the PyTorch convention), the same computation is one matrix product per layer:

which is exactly what nn.Linear(n_in, n_out) does: it stores $\mathbf{W}$ as an $n_{out}\times n_{in}$ matrix and computes $\mathbf{X}\mathbf{W}^\top+\mathbf{b}$ with broadcasting (2.2 NumPy and vectorized thinking). Batching is why GPUs love neural networks: one big matmul instead of $B$ small ones (4.12 Compute, hardware and scale basics).

Parameter count. Layer $\ell$ has $n_\ell n_{\ell-1}$ weights and $n_\ell$ biases, so

For an MNIST classifier $784\to256\to128\to10$: $(784+1)\cdot256 = 200{,}960$, $(256+1)\cdot128 = 32{,}896$, $(128+1)\cdot10 = 1{,}290$, total $235{,}146$. The first layer holds 85% of the weights — a pattern you will see again when counting transformer parameters (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).

Why the nonlinearity is not optional. Compose two affine maps and you get another affine map:

Without $\phi$, a 100-layer network is a single linear model with extra steps and still cannot do XOR. The nonlinearity is where the expressive power comes from; which one to choose is the subject of 4.2 Activation functions.

The universal approximation theorem — and what it does not say

Theorem (Cybenko 1989; Hornik 1991). Let $\phi$ be a continuous, non-polynomial activation and $K\subset\mathbb{R}^d$ compact. For every continuous $f:K\to\mathbb{R}$ and every $\varepsilon>0$ there exist a width $N$ and parameters such that the one-hidden-layer network $g(\mathbf{x})=\sum_{i=1}^{N} v_i\,\phi(\mathbf{w}_i^\top\mathbf{x}+b_i)$ satisfies $\sup_{\mathbf{x}\in K}|f(\mathbf{x})-g(\mathbf{x})|<\varepsilon$.

Intuition. In one dimension, the difference of two shifted ReLUs, $\mathrm{ReLU}(x-a)-\mathrm{ReLU}(x-b)$, is a ramp that rises from $a$ to $b$ and then stays flat. Sums of ramps make any piecewise-linear function, and piecewise-linear functions approximate any continuous function on an interval as finely as you like. Each hidden unit contributes one "kink".

The theorem is reassuring but routinely over-interpreted. It does not say:

1. that the width is reasonable — for some functions the required $N$ grows exponentially with the input dimension $d$;
2. that gradient descent will find those weights — existence of a solution is not learnability from finite data with SGD;
3. anything about generalization — a network that fits the training set perfectly may fail on new inputs (3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 4.7 Regularization in deep nets);
4. that depth is pointless — deep ReLU networks can represent certain compositional functions with exponentially fewer units than any shallow network (the number of linear regions a ReLU net can carve grows exponentially with depth but only polynomially with width).

Depth, width and what layers do

Each layer re-represents its input, so a deep network builds a hierarchy of features: in images, edges become textures become parts become objects (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt); in language, characters become morphemes become syntax become meaning (7.6 Decoder-only: GPT and autoregressive generation). Depth is composition, and composition is cheap to express and hard to match with width. The catch is that very deep plain MLPs are hard to optimize — gradients shrink or explode as they pass through many layers (4.2 Activation functions), so careful initialization (4.6 Initialization and normalization) and residual connections (5.3 Architectures: LeNet → AlexNet → VGG → Inception → ResNet → EfficientNet → ConvNeXt, 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings) were needed before depth became practical. Today's MLPs are mostly two-layer blocks living inside larger architectures: the feed-forward sub-layer of a transformer is literally $\mathbf{W}_2\,\phi(\mathbf{W}_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2$ applied to each token independently.

The geometry of a ReLU network

Because ReLU is linear on each side of zero, a ReLU network is a piecewise-linear function: the input space is cut into polyhedral regions, and inside each region the network is a single affine map. Decision boundaries are therefore made of flat facets, and more units or layers mean more (and smaller) facets, which is why the nn-playground demo below shows jagged boundaries for small networks and smooth-looking ones for larger networks. The same picture explains why networks can be confidently wrong far from the data: outside the training region the function keeps extrapolating linearly.

In code

import torch, torch.nn as nn
model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(784, 256), nn.ReLU(),
    nn.Linear(256, 128), nn.ReLU(),
    nn.Linear(128, 10),           # logits; the loss applies softmax
)
print(sum(p.numel() for p in model.parameters()))   # 235146
x = torch.randn(32, 1, 28, 28)                        # a batch of 32 images
print(model(x).shape)                                 # torch.Size([32, 10])

Where this shows up later

• The nonlinearities that make depth work and the vanishing-gradient problem — 4.2 Activation functions; the losses attached to the logits — 4.3 Loss functions catalog.
• Computing gradients through this exact stack — 4.4 Backpropagation, fully derived; keeping activations well-scaled through many layers — 4.6 Initialization and normalization.
• A convolution is an MLP layer with shared, local weights — 5.1 Convolution: the operation and its math; the transformer's feed-forward block is a two-layer MLP per token — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings.
• Logistic regression is the zero-hidden-layer special case — 3.3 Logistic regression and softmax classification; a Q-network is an MLP from states to action values — 4.11 Reinforcement learning essentials.