Beyond vanilla attention: state space models, long context and hybrids

The quadratic wall

Attention gives every token a direct line to every other, at $O(n^2 d)$ compute per layer and a KV cache that grows linearly with context (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization). At 128k tokens the attention term dominates the forward pass (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings); at 1M tokens a single Llama-3-8B-style KV cache is 128 GB. Three families of answers exist: make attention cheaper (sparse, windowed, linear), replace it with a recurrence that trains in parallel (state space models), or mix the two (hybrids). A fourth answer — retrieve only what you need — is the subject of Phase 9.

Cheaper attention

Sliding-window attention (Mistral 7B, Gemma's local layers) lets each token attend only to the previous $w$ tokens (for example 4,096). Cost per layer drops to $O(n\,w\,d)$ and the KV cache to a rolling buffer of $w$ entries. Information still travels far because the receptive field grows by $w$ per layer — after $L$ layers a token can be influenced by $L\cdot w$ positions — but precise long-range lookup degrades. Many 2025–26 models alternate local and global attention layers to get most of the savings while keeping some exact global access.

Linear attention removes the softmax so that associativity can be exploited. With a feature map $\phi$,

because $\phi(\mathbf{K})^\top\mathbf{V}$ is a $d\times d$ matrix computed once. Causally, this becomes a recurrence $\mathbf{S}_t = \mathbf{S}_{t-1} + \phi(\mathbf{k}_t)\mathbf{v}_t^\top$, $\ \mathbf{y}_t = \phi(\mathbf{q}_t)^\top\mathbf{S}_t$: a fixed-size state instead of a growing cache. The catch is quality — a $d\times d$ state cannot store an arbitrary number of exact key–value pairs, which is precisely what softmax attention does well.

State space models

An SSM starts from a continuous linear system, $\mathbf{h}'(t) = \mathbf{A}\mathbf{h}(t) + \mathbf{B}x(t)$, $y(t) = \mathbf{C}\mathbf{h}(t)$, and discretizes it with step $\Delta$ into a recurrence

Because the recurrence is linear, unrolling it gives $y_t = \sum_{s\le t} \mathbf{C}\bar{\mathbf{A}}^{\,t-s}\bar{\mathbf{B}}\,x_s$ — a convolution with a kernel determined by $(\bar{\mathbf{A}}, \bar{\mathbf{B}}, \mathbf{C})$. S4 (2021) exploited this duality: train in parallel as a long convolution (via FFT, $O(n\log n)$), infer as a recurrence with $O(1)$ cost and constant memory per step, with a structured $\mathbf{A}$ (the HiPPO initialization) that remembers long histories. S4 excelled on long-range benchmarks but lagged transformers on language.

Mamba (2023) diagnosed why: with fixed $\mathbf{A}, \mathbf{B}, \mathbf{C}$ the model cannot decide what to remember based on content. Mamba makes $\mathbf{B}$, $\mathbf{C}$ and $\Delta$ functions of the input $x_t$ — a selective SSM — so a token can open or close the memory gate the way an LSTM (6.5 LSTM and GRU) does. Selectivity breaks the convolution trick, so Mamba computes the recurrence with a hardware-aware parallel scan that keeps the state in fast SRAM. Result: linear time in sequence length, constant inference memory, and language-modelling quality matching transformers of the same size at the scales tested. Mamba-2 (2024) showed that selective SSMs are a form of structured masked attention — the state space duality — enabling matrix-multiplication-based implementations and borrowing transformer tricks. RWKV reached similar goals from the RNN side with a time-mixing operator that trains in parallel.

Intuition. Attention keeps a verbatim transcript and re-reads it for every new word. An SSM keeps a running summary of fixed size. Mamba lets the model decide, word by word, what to write into the summary and what to let fade.

Where recurrence struggles

A fixed-size state is a compressed summary. Tasks that need exact retrieval of an arbitrary earlier token — copying a long string, looking up a phone number from page 3, multi-needle "needle in a haystack" tests, in-context learning over many examples — favour attention, whose cache stores everything verbatim. This is why pure SSMs lag on retrieval-heavy evaluations, and why the practical answer became hybrids.

Hybrids: the efficiency default in 2026

Interleave a few attention layers among many SSM or linear-attention layers: Jamba (Mamba + attention + MoE), Griffin/RecurrentGemma, Zamba and several newer models follow this pattern. The attention layers supply exact recall; the recurrent layers supply cheap long-range context. Typical ratios use one attention layer per 4–8 recurrent layers, cutting the KV cache by a similar factor while matching transformer quality on most benchmarks.

Long context in transformers

Instead of changing the mechanism, you can stretch the transformer: RoPE scaling and YaRN (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi) extend positions; ring attention splits a sequence across devices and passes K/V blocks around a ring so that a million tokens fit in aggregate memory; block-sparse and dilated patterns and Infini-attention-style compressive memories bound the cost. Evaluate with needle-in-a-haystack and multi-hop long-document tasks, and remember "lost in the middle" (9.6 Generation side: context assembly, citations, structured outputs and tool calling): a model that accepts 1M tokens does not necessarily use them.

What is settled in 2026

Softmax attention remains the core of frontier models because exact recall and in-context learning matter most; GQA, MoE and FlashAttention made it affordable. Linear and SSM layers are mainstream as components of hybrids, especially for long-context and on-device inference, and the SSM–attention duality has turned "attention versus recurrence" into a design spectrum rather than a contest. Long context and retrieval are complements, not substitutes (9.1 Why RAG, and the canonical pipeline).

Where this shows up later

• Long context versus retrieval, and lost-in-the-middle effects — 9.1 Why RAG, and the canonical pipeline, 9.6 Generation side: context assembly, citations, structured outputs and tool calling.
• Hybrid and MoE architectures shape the model landscape and its costs — 8.1 The LLM lifecycle and model landscape; serving costs — 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization.
• Recurrent memory and compaction reappear as agent memory design, where the context window is the working memory — 10.4 Memory and context engineering.