Encoder-only: BERT and friends (and sentence embeddings)

Reading in both directions

A decoder predicts the next token, so position $t$ may only see positions $< t$. An encoder has no such constraint: every token attends to every other, left and right. That makes encoders useless for generation but ideal for understanding a fixed piece of text — classifying it, tagging each token, extracting an answer span, or compressing it into a vector. BERT (Devlin et al., 2018) is the transformer encoder of 7.1 Attention Is All You Need: the big picture trained with a self-supervised objective that exploits bidirectionality.

Masked language modelling

You cannot train a bidirectional model to predict the next word — it can already see it. BERT's solution is the masked LM (MLM): corrupt 15% of input tokens and predict the originals from the two-sided context. Of the selected tokens, 80% are replaced by a special [MASK] token, 10% by a random token, and 10% are left unchanged, so that the model cannot rely on seeing [MASK] at inference (where it never appears) and must keep a good representation of every token. The loss is cross-entropy on the masked positions only:

where $\mathcal{M}$ is the set of corrupted positions and $\tilde{\mathbf{x}}$ the corrupted sequence. Note the cost: only 15% of tokens produce a training signal per pass, versus 100% for a causal LM (7.6 Decoder-only: GPT and autoregressive generation) — one reason BERT-style pretraining fell behind at scale.

The original BERT added next-sentence prediction (NSP), a binary task on sentence pairs separated by [SEP]; RoBERTa (2019) showed NSP was unnecessary and that BERT was simply under-trained: more data (160 GB vs 16 GB), longer training, bigger batches and dynamic masking (a fresh mask each epoch) gave large gains with the identical architecture. Input embeddings are the sum of token, position and segment embeddings; the vocabulary is 30k WordPiece tokens (6.1 Text to numbers: tokenization). Sizes: BERT-base has $L=12$, $d=768$ and 110M parameters; BERT-large has $L=24$, $d=1024$ and 340M.

Fine-tuning: one encoder, many heads

Pretraining produces contextual vectors for every token. Fine-tuning attaches a small head and trains everything end to end for a few epochs at a low learning rate (8.2 Fine-tuning and PEFT):

This "pretrain once, fine-tune cheaply" recipe (ELMo and ULMFiT pioneered it, 6.3 Word embeddings) made BERT the default NLP tool from 2019 to 2022, and it still powers countless production classifiers because a 110M-parameter encoder is cheap to run.

Better encoders

• ELECTRA (2020) replaces MLM with replaced-token detection: a small generator fills masks with plausible tokens and the main model classifies every token as original or replaced. Now 100% of positions give signal, so ELECTRA matches RoBERTa with far less compute.
• DeBERTa separates content and position in attention (disentangled attention) and was the strongest GLUE/SuperGLUE encoder of its era.
• ModernBERT (2024) retrofits the encoder with the decoder-world toolkit — RoPE (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi), GeGLU, FlashAttention (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization), an 8k context and 2T training tokens — showing that the encoder family had been neglected, not made obsolete.

From tokens to sentence embeddings

RAG needs one vector per passage (9.2 Embeddings deep dive). The naive choice — BERT's [CLS] vector or the mean of its token vectors — gives poor similarity scores, because MLM never asked for sentence-level geometry. Sentence-BERT (2019) fixes this with a siamese setup: the same encoder embeds two sentences, mean pooling gives $\mathbf{u}$ and $\mathbf{v}$, and the model is fine-tuned so that cosine similarity (1.1 Vectors, dot products and norms) tracks semantic similarity, using NLI pairs or a contrastive loss that pulls positives together and pushes in-batch negatives apart (5.8 Self-supervised and contrastive learning). Modern embedding models (E5, BGE, GTE, Nomic) are exactly this — encoder transformers trained contrastively on hundreds of millions of pairs — and a cross-encoder (both texts in one input, score read from [CLS]) is the slower, more accurate reranker applied to the top-$k$ results (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics).

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')          # a 22M-parameter BERT-style encoder
emb = model.encode(['How do I reset my password?',
                    'Password recovery steps',
                    'Best pizza in Rome'], normalize_embeddings=True)
print(emb @ emb.T)      # cosine similarities: first two high, third low

Intuition. MLM teaches an encoder what each word means in context; contrastive fine-tuning teaches it what a whole passage is about. Retrieval needs the second.

Encoder-only in 2026

Use an encoder when you need a representation rather than text: embeddings for retrieval and clustering, high-volume classification (moderation, routing, intent), token tagging, or reranking. A 100M–400M-parameter encoder processes thousands of passages per second on one GPU, orders of magnitude cheaper than prompting a decoder LLM for the same label — though zero-shot LLM classification wins when you have no labelled data at all (8.4 Prompting as programming).

Where this shows up later

• Bi-encoders and cross-encoders trained from these models are the retrievers and rerankers of RAG — 9.2 Embeddings deep dive, 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics.
• Masked modelling returns as MAE for images (5.8 Self-supervised and contrastive learning) and as span corruption in T5 (7.7 Encoder–decoder: T5, BART and choosing an architecture).
• Fine-tuning with a small head is the simplest form of adaptation; LoRA generalizes it — 8.2 Fine-tuning and PEFT.
• Contrastive training with in-batch negatives (InfoNCE) is the same objective behind SimCLR and CLIP — 5.8 Self-supervised and contrastive learning, 8.10 Multimodal models.