Encoder–decoder: T5, BART and choosing an architecture

Two stacks, one bridge

An encoder–decoder keeps both halves of the 2017 design (7.1 Attention Is All You Need: the big picture): a bidirectional encoder reads the input once, and a causal decoder generates the output while attending to the encoder's final states through cross-attention — the attention function of 7.2 Scaled dot-product and multi-head attention, fully derived with queries from the decoder and keys/values from the encoder. Nothing forces input and output to share a vocabulary, a modality or even a length: the encoder can consume audio frames or image patches while the decoder writes text.

T5: everything is text-to-text

T5 (Raffel et al., 2019) asked a simple question — if every NLP task is cast as "text in, text out", which pretraining choices matter? Translation becomes "translate English to German: …", classification becomes generating the label word, and even regression becomes generating a number as a string. The paper's systematic study (architectures, objectives, data, scale; 11B parameters at the top) settled several things:

• Span corruption beat plain masking and next-token prediction for this setup: random spans (mean length 3, 15% of tokens) are replaced by sentinel tokens, and the decoder must output the dropped spans, each prefixed by its sentinel. For example, the input "Thank you <X> me to your party <Y> week." has the target "<X> for inviting <Y> last <Z>". The loss is ordinary cross-entropy on the short target, so the decoder works on far fewer tokens than the encoder reads — cheap.
• Clean data matters: the Colossal Clean Crawled Corpus (C4) filtered Common Crawl with simple heuristics and became a standard dataset.
• Relative position biases (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi) replaced sinusoids, and pre-norm with RMSNorm replaced post-norm LayerNorm (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).
• Instruction-tuned successors (Flan-T5, 2022) made T5 one of the strongest open models of its size, and the T5 encoder lives on as the text encoder inside image-generation systems such as Imagen and Stable Diffusion 3 (8.9 Generative models II: Diffusion and flow matching, 8.10 Multimodal models).

BART: a denoising autoencoder for text

BART (Lewis et al., 2019) uses the standard seq2seq transformer and pretrains it to reconstruct text from corrupted text. The corruptions are token masking, token deletion, text infilling (a whole span replaced by a single mask, so the model must also guess the span's length), sentence permutation and document rotation; the decoder is trained to emit the original document. Because the decoder reproduces the whole input, BART is particularly strong at summarization and other generation tasks conditioned on a document, and it remains a common baseline there.

Intuition. BERT learns to fill holes, GPT learns to continue, T5 and BART learn to rewrite: read something, then produce something else that depends on all of it.

Where encoder–decoders still win

The architecture shines when the input is large and fixed and the output is short: the encoder processes the input once, bidirectionally, and the decoder only generates the answer.

• Speech: Whisper (8.10 Multimodal models) encodes 30 seconds of audio spectrogram and decodes text; the modality gap makes two stacks natural.
• Translation: dedicated systems (NLLB, MADLAD) remain encoder–decoders; they are small, fast and accurate on their task.
• Efficiency on long inputs: for a 2,000-token document and a 50-token answer, a decoder-only model runs every generation step over all 2,050 positions, while an encoder–decoder's decode steps attend over 50 target positions plus cross-attention into cached encoder states.

Where they lose: a decoder-only model sees prompt and answer in one stream, which makes in-context learning, multi-turn dialogue and open-ended generation simpler, and its scaling behaviour is better understood (7.9 Scaling laws and compute-optimal training). The prefix LM — a decoder with bidirectional attention over the prompt and causal attention over the answer — is a middle ground found in some models.

Choosing an architecture

A rule of thumb: need a vector → encoder; need open-ended text or reasoning → decoder; need to map one fixed input to a short output across a modality gap → encoder–decoder.

Cross-attention in code

Cross-attention reuses the attention function from 7.2 Scaled dot-product and multi-head attention, fully derived with a different source for keys and values:

class CrossAttention(torch.nn.Module):
    def __init__(self, d, n_heads):
        super().__init__()
        self.h, self.d_k = n_heads, d // n_heads
        self.q = torch.nn.Linear(d, d, bias=False)
        self.kv = torch.nn.Linear(d, 2 * d, bias=False)
        self.out = torch.nn.Linear(d, d, bias=False)

    def forward(self, x_dec, enc):              # x_dec: (B, n_t, d), enc: (B, n_s, d)
        B, n_t, d = x_dec.shape
        n_s = enc.size(1)
        q = self.q(x_dec).view(B, n_t, self.h, self.d_k).transpose(1, 2)
        k, v = self.kv(enc).split(d, dim=-1)
        k = k.view(B, n_s, self.h, self.d_k).transpose(1, 2)
        v = v.view(B, n_s, self.h, self.d_k).transpose(1, 2)
        z, _ = attention(q, k, v, causal=False)  # every source position is visible
        return self.out(z.transpose(1, 2).reshape(B, n_t, d))

The score matrix is $n_t\times n_s$ — target positions by source positions — which is exactly the alignment matrix of classic attention (6.7 Attention is born). Since the encoder output does not change during decoding, its projected keys and values are computed once and cached, which is why decode steps are cheap.

Where this shows up later

• Whisper's audio encoder and text decoder — 8.10 Multimodal models; T5's encoder as the text conditioner in diffusion models — 8.9 Generative models II: Diffusion and flow matching, 8.10 Multimodal models.
• Cross-attention is also how image-generation U-Nets and DiTs inject text conditioning — 8.9 Generative models II: Diffusion and flow matching.
• The decision table reappears when you pick a retriever (encoder), a generator (decoder) and a reranker (cross-encoder) for RAG — 9.2 Embeddings deep dive, 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics.