Recurrent neural networks and BPTT

Sequences need shared weights

A feed-forward network (4.1 From perceptron to MLP; universal approximation) takes a fixed-size input. Text, audio and time series have variable length and the same patterns can appear at any position. Karpathy's taxonomy of sequence tasks: one-to-one (image classification), one-to-many (image captioning), many-to-one (sentiment, "is this review positive?"), many-to-many aligned (part-of-speech tagging) and unaligned (translation, 6.6 Seq2seq, encoder–decoder and beam search). A recurrent network handles all of them with one idea: process the sequence one step at a time, carrying a hidden state that summarizes everything seen so far, and reuse the same weights at every step.

The recurrence

With input $\mathbf{x}_t\in\mathbb{R}^{d}$ (an embedding, 6.3 Word embeddings) and hidden state $\mathbf{h}_t\in\mathbb{R}^{n}$,

The parameter count is $n^2+nd+n$ for the recurrence plus $nV+V$ for the output — independent of sequence length. Unrolling the recurrence over $T$ steps gives a $T$-layer-deep network whose layers share weights; that picture is what makes training possible with ordinary backprop (4.4 Backpropagation, fully derived). For a language model the target at step $t$ is the next token $w_{t+1}$ and the loss is the cross-entropy summed over steps, $L=\sum_t L_t$, exactly the n-gram objective of 6.2 Classical NLP and n-gram language models with an unbounded context. A char-RNN trained on Shakespeare generates plausible Elizabethan English by sampling one character at a time and feeding it back in.

Backpropagation through time

Apply the chain rule to the unrolled graph. The loss at step $t$ depends on $\mathbf{W}_{hh}$ through every earlier hidden state, so

where $\partial^{+}$ denotes the "immediate" derivative treating $\mathbf{h}_{k-1}$ as a constant. Each factor $\frac{\partial\mathbf{h}_j}{\partial\mathbf{h}_{j-1}}$ is the Jacobian of one step: the recurrent weight matrix times the diagonal of activation derivatives (for $\tanh$, $1-\tanh^2(\mathbf{a}_j)\in(0,1]$). Because weights are shared, the gradient of the total loss is the sum of these contributions over all $t$ and all $k\le t$, and in code you simply run reverse-mode autograd over the unrolled graph — that is BPTT.

Intuition. The term $\frac{\partial\mathbf{h}_t}{\partial\mathbf{h}_k}$ says how much the state at time $t$ still "remembers" a nudge made at time $k$. For the network to learn a long-range dependency (a subject at position 3 fixing a verb at position 40), this term must be reasonably large for $t-k=37$.

Vanishing and exploding gradients

Bound the norm of the Jacobian product using sub-multiplicativity of norms (1.2 Matrices as transformations): if $\|\mathbf{W}_{hh}^\top\|\le\lambda$ (its largest singular value) and $\|\mathrm{diag}(\tanh')\|\le\gamma$ (with $\gamma=1$ for $\tanh$, $\gamma=1/4$ for the sigmoid), then

If $\lambda\gamma<1$ the bound decays exponentially — the gradient from distant steps vanishes and the model learns only short-range structure. If $\lambda\gamma>1$ the gradient can explode, producing NaNs and wrecked parameters. In one dimension the picture is exact: with recurrent weight $0.9$ a signal from 50 steps back is scaled by $0.9^{50}\approx0.005$; with weight $1.1$ it is scaled by $1.1^{50}\approx117$. (Pascanu, Mikolov & Bengio, 2013, make this rigorous with the largest eigenvalue of $\mathbf{W}_{hh}$.) Saturated $\tanh$ units make things worse because $\tanh'\approx0$ there — the same saturation problem as in deep feed-forward nets (4.2 Activation functions).

Remedies, in order of importance:

• Gradient clipping for explosions: if $\|\mathbf{g}\|>\theta$, rescale $\mathbf{g}\leftarrow\theta\,\mathbf{g}/\|\mathbf{g}\|$. Cheap, standard, used for transformers too (4.5 Optimizers and learning-rate schedules).
• Gating for vanishing: LSTM and GRU add an additive path for the state so the Jacobian has a factor close to the identity (6.5 LSTM and GRU). Residual connections are the feed-forward analogue (4.6 Initialization and normalization).
• Initialization and architecture: orthogonal or identity initialization of $\mathbf{W}_{hh}$ (singular values near 1), ReLU RNNs with identity init, layer normalization inside the recurrence.
• Truncated BPTT for cost, not gradients: unroll only $k$ steps backward while carrying the hidden state forward. It bounds memory and compute but also bounds the dependency length the model can learn.

Training RNNs in practice

Use teacher forcing: feed the true previous token as input during training, not the model's own sample (6.6 Seq2seq, encoder–decoder and beam search). Batch variable-length sequences with padding and a mask so padded positions contribute no loss (or pack sequences). Apply dropout to inputs/outputs and between layers but not to the recurrent connection unless you use variational dropout (the same mask at every step). Monitor perplexity on held-out text (6.2 Classical NLP and n-gram language models). At inference, sample with a temperature (8.5 Decoding and sampling): low temperature gives repetitive, safe text; high temperature gives inventive nonsense. Sequential dependence means an RNN cannot parallelize over time during training — each step waits for the previous hidden state — which is the computational limitation that attention removed (7.1 Attention Is All You Need: the big picture).

Where this shows up later

• Gated recurrences — 6.5 LSTM and GRU; encoder–decoder RNNs and attention — 6.6 Seq2seq, encoder–decoder and beam search, 6.7 Attention is born.
• The vanishing-gradient product motivates residual connections and normalization — 4.6 Initialization and normalization; clipping and schedules — 4.5 Optimizers and learning-rate schedules.
• Transformers replace the $O(T)$ sequential path with $O(1)$ attention paths — 7.1 Attention Is All You Need: the big picture; linear RNNs and state-space models revive the recurrence with parallel training — 7.12 Beyond vanilla attention: state space models, long context and hybrids; the character-level LM project — 6.8 Project: character-level language model.