Interpretability: what is happening inside

The question

Saliency maps (5.9 Interpretability, robustness and adversarial examples) tell you which inputs mattered; mechanistic interpretability asks what algorithm the weights implement. The ambition is to reverse-engineer a network into human-understandable parts — circuits built from features — the way one reverse-engineers a compiled program.

The residual stream and the logit lens

In a transformer (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings) every layer reads from and writes to the same vector per token — the residual stream. Attention heads move information between positions; MLPs transform it in place; the final unembedding reads the stream as logits. Because every layer writes in the same coordinate system, you can apply the final LayerNorm and unembedding to an intermediate residual state — the logit lens — and watch the model's prediction form layer by layer: early layers guess syntax, middle layers settle facts, late layers refine. Tuned variants (fitting a small affine map per layer) make the readout more reliable.

Attention heads as circuits

Decompose a head into two circuits (7.2 Scaled dot-product and multi-head attention, fully derived). The QK circuit $\mathbf{W}_Q^\top\mathbf{W}_K$ decides where to attend (which source positions match the query); the OV circuit $\mathbf{W}_O\mathbf{W}_V$ decides what to copy from the attended position into the residual stream. Heads compose across layers: a later head's query can depend on what an earlier head wrote.

The canonical example is the induction head. If the context contains the pattern [A][B] … [A], the model predicts [B]. Two heads cooperate: a previous-token head in an earlier layer writes "my predecessor was A" into each position's residual stream; the induction head in a later layer uses that to attend from the current [A] to the token after the earlier [A], then copies it. Induction heads appear abruptly during training as a phase change, coincide with a sharp drop in loss, and account for much of in-context learning (8.4 Prompting as programming) — including on patterns never seen in training. More complex circuits (indirect-object identification, copy suppression) have been mapped in small models.

MLPs, neurons and superposition

MLP layers behave like key–value memories (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings): the first matrix matches patterns in the residual stream, the second writes associated vectors — many factual associations live here, which is why editing a few MLP weights can change a fact. But individual neurons are usually polysemantic — one neuron fires for academic citations, Korean text and a specific HTTP header. The explanation is superposition: the model represents more features than it has dimensions by assigning them nearly orthogonal directions (recall from 1.1 Vectors, dot products and norms that high-dimensional space holds exponentially many almost-orthogonal vectors). Interference is tolerable because features are sparse — few are active at once. Toy models (Anthropic, 2022) show superposition appearing exactly when features are sparse and the model is capacity-limited.

Sparse autoencoders: pulling features apart

If features are sparse directions in superposition, a dictionary-learning problem recovers them. Train a wide autoencoder on residual-stream activations $\mathbf{x}$:

with far more latent units than input dimensions and an $\ell_1$ (or top-$k$) sparsity penalty (3.6 Regularization and generalization — the same geometry that gives lasso its zeros). The learned decoder columns are candidate features; many turn out to be monosemantic and human-interpretable — a feature for the Golden Gate Bridge, for code bugs, for sycophantic praise. Scaled to production-size models (Scaling Monosemanticity, 2024), SAEs found millions of features, and clamping one changes behavior predictably (the model that could not stop talking about the Golden Gate Bridge). Limitations are real: reconstruction is imperfect, features split or merge as the dictionary size changes, some latents are dead, and "interpretable" is judged by humans or LLMs with their own biases.

Probing, steering and attribution graphs

• Linear probes train a classifier on activations to detect a concept (truth vs falsehood, position in a board game). A good probe shows the information is linearly present — not that the model uses it.
• Activation steering adds a direction to the residual stream at inference (a "refusal direction", a sentiment vector) and shifts behavior without retraining — evidence that features are causal, and a potential control tool.
• Attribution graphs (2025) use replacement models built from features to trace how a specific output was produced across layers and tokens. Case studies showed models planning rhymes several tokens ahead, performing multi-step factual reasoning internally, sharing circuits across languages — and producing chains of thought that did not match the actual computation, a direct check on the faithfulness worries of 8.6 Reasoning models and test-time compute.

What we know, what we don't

Established: circuits exist and can be mapped in small models; features are directions, often in superposition; SAEs recover many interpretable features; models sometimes plan ahead; stated reasoning is not always faithful. Open: no frontier model is close to fully understood; SAE features explain only part of the variance; methods scale poorly in human effort; and using interpretability as a reliable safety audit (e.g. detecting deception) is early-stage. The debate between "ambitious" mechanistic interpretability and pragmatic, top-down methods (probes, steering, behavioral evals) is ongoing and healthy.

How to start

Tools: TransformerLens for cached activations and hooks, SAELens and public SAE dictionaries, Neuronpedia for browsing features. Curricula: Neel Nanda's quickstart guide and the ARENA program's interpretability track. The classic entry exercise is to find induction heads in a two-layer attention-only model yourself.

Where this shows up later

• Faithfulness of reasoning and the limits of alignment — 8.6 Reasoning models and test-time compute, 8.12 Safety, risks and responsible AI.
• Residual-stream and attention mechanics — 7.2 Scaled dot-product and multi-head attention, fully derived, 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; the sparse geometry — 1.1 Vectors, dot products and norms, 3.6 Regularization and generalization.
• Interpretability as a research direction — 11.6 Frontier 2026: where the field is heading.