Word embeddings

The distributional hypothesis

"You shall know a word by the company it keeps" (Firth, 1957). Words that appear in similar contexts tend to have similar meanings: tea and coffee both follow "a cup of" and precede "with milk". If we can turn contexts into numbers, we can turn meaning into geometry.

From one-hot to dense

A one-hot vector for word $w$ is a $V$-dimensional vector with a single 1. Every pair of one-hot vectors is orthogonal (1.1 Vectors, dot products and norms), so they encode identity and nothing about similarity: $\mathbf{e}_{\text{cat}}\cdot\mathbf{e}_{\text{dog}}=0$. A dense embedding maps each word to a vector in $\mathbb{R}^d$ with $d\ll V$ (50–300 for word vectors, 768–4096 inside LLMs) in which similar words have high cosine similarity. The embeddings live in a matrix $\mathbf{E}\in\mathbb{R}^{V\times d}$, and "looking up" word $w$ is the matrix–vector product $\mathbf{E}^\top\mathbf{e}_w$ — multiplying by a one-hot vector just selects a row (1.2 Matrices as transformations). The rows are learned parameters, trained by gradient descent like any other weight (4.4 Backpropagation, fully derived).

word2vec: the skip-gram model

Mikolov et al. (2013) trained embeddings with a deliberately simple task: given a center word $c$, predict each word $o$ within a window of $\pm m$ positions. Each word gets two vectors, $\mathbf{v}_w$ as a center and $\mathbf{u}_w$ as a context, and the prediction is a softmax over the vocabulary:

This is a one-layer neural network with cross-entropy loss (4.3 Loss functions catalog): the hidden layer is the embedding. The dot product $\mathbf{u}_o^\top\mathbf{v}_c$ is high when the context vector of $o$ aligns with the center vector of $c$ — the alignment-as-similarity idea from 1.1 Vectors, dot products and norms. Its problem is the denominator: a sum over the whole vocabulary for every training pair, $O(V)$ per update.

Negative sampling

Skip-gram with negative sampling (SGNS) replaces the $V$-way softmax with a binary task: is this (center, context) pair real or noise? For each observed pair $(c,o)$ draw $K$ noise words $w_k$ from a noise distribution $P_n$ and maximize

where $\sigma$ is the sigmoid. The first term pushes real pairs together; the second pushes the center away from $K$ random words (5–20 of them). Each update now costs $O(Kd)$ instead of $O(Vd)$. The noise distribution is the unigram distribution raised to the $3/4$ power, $P_n(w)\propto U(w)^{3/4}$, which samples rare words more often than their frequency alone would. Frequent words ("the") are also subsampled from the training text so they do not dominate.

Math check. Treat the two terms as logistic regression (3.3 Logistic regression and softmax classification) with label 1 for the real pair and 0 for noise pairs. Using $\frac{d}{dx}\log\sigma(x)=1-\sigma(x)$ and $\frac{d}{dx}\log\sigma(-x)=-\sigma(x)$, the gradient of the per-pair objective with respect to the center vector is

Gradient ascent moves $\mathbf{v}_c$ toward the true context vector (by how surprised the model was) and away from each noise vector (by how much it wrongly believed them). The same "pull positives, push negatives" structure is the InfoNCE loss of contrastive learning (5.8 Self-supervised and contrastive learning, 9.2 Embeddings deep dive).

CBOW (continuous bag of words) reverses the task — predict the center from the average of the context vectors — and trains faster; skip-gram is better for rare words. Levy and Goldberg later showed that SGNS implicitly factorizes a shifted pointwise-mutual-information matrix of word co-occurrences, connecting the neural method back to counting (6.2 Classical NLP and n-gram language models).

GloVe: regression on co-occurrence counts

GloVe (Pennington, Socher & Manning, 2014) starts from the global co-occurrence matrix $X_{ij}$ (how often word $j$ appears in the context of word $i$) and observes that ratios of co-occurrence probabilities carry meaning: $P(\text{solid}\mid\text{ice})/P(\text{solid}\mid\text{steam})$ is large, $P(\text{gas}\mid\text{ice})/P(\text{gas}\mid\text{steam})$ is small, and $P(\text{water}\mid\cdot)$ gives a ratio near 1. Requiring dot products to model log counts yields a weighted least-squares objective:

where the weighting $f$ ignores zero counts and caps the influence of very frequent pairs. GloVe trains on the (sparse) count matrix rather than on the text stream, and its word vectors behave much like word2vec's.

fastText: words as bags of character n-grams

word2vec and GloVe assign one vector per word type, so a word unseen in training has no vector and run/running/runner share nothing. fastText represents a word as the sum of the vectors of its character n-grams (3- to 6-grams, plus the whole word): where → ⟨wh, whe, her, ere, re⟩, ⟨where⟩. Morphologically related words share pieces, rare words borrow strength from their parts, and any new word gets a vector. It is the same subword insight as BPE (6.1 Text to numbers: tokenization) applied to embeddings. fastText also ships a very fast text classifier built on averaged embeddings.

Analogies, evaluation and bias

The famous result: $\mathbf{v}_{\text{king}}-\mathbf{v}_{\text{man}}+\mathbf{v}_{\text{woman}}\approx\mathbf{v}_{\text{queen}}$ (nearest neighbour by cosine, excluding the input words). Directions encode relations: gender, tense, capital-of, comparative. The effect is real but fragile — many analogies fail, and the standard evaluation quietly excludes the three input words. Intrinsic evaluations (word-similarity datasets, analogy sets) correlate only loosely with extrinsic performance on a downstream task, which is what you should care about.

Embeddings inherit the statistics of their corpus, including its stereotypes: Bolukbasi et al. (2016) found man : computer programmer :: woman : homemaker in Google News vectors, and occupations aligned along a gender direction. "Debiasing" by projecting out that direction removes the obvious analogy while leaving the bias recoverable from the geometry — a warning that applies to LLMs (8.12 Safety, risks and responsible AI).

From static to contextual

A static embedding gives bank one vector for riverbanks and savings accounts. The fix was to make the vector depend on the sentence: ELMo (2018) used the hidden states of a bidirectional LSTM language model (6.5 LSTM and GRU), and BERT (2019) used a transformer (7.5 Encoder-only: BERT and friends (and sentence embeddings)), producing contextual embeddings that differ for every occurrence. Modern sentence and document embeddings for search (9.2 Embeddings deep dive) are contextual models fine-tuned with the same contrastive idea as negative sampling. Yet the first layer of every one of these models is still a static lookup table $\mathbf{E}$ (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings) — word2vec's descendant.

Where this shows up later

• Embedding tables and weight tying in transformers — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; contextual embeddings from BERT — 7.5 Encoder-only: BERT and friends (and sentence embeddings); sentence embeddings and MTEB — 9.2 Embeddings deep dive.
• Negative sampling → InfoNCE and contrastive learning — 5.8 Self-supervised and contrastive learning, CLIP — 8.10 Multimodal models; cosine similarity and high-dimensional geometry — 1.1 Vectors, dot products and norms.
• Bias and fairness in learned representations — 8.12 Safety, risks and responsible AI; probing what directions in activation space mean — 8.11 Interpretability: what is happening inside.