Vectors, dot products and norms

Two pictures of a vector

A vector $\mathbf{x} \in \mathbb{R}^n$ is an ordered list of $n$ real numbers, written as a column:

Keep two pictures in mind at once. Geometrically, $\mathbf{x}$ is an arrow from the origin to a point. As data, it is one row of a spreadsheet: a house is $(\text{area}, \text{rooms}, \text{age})$, a 28×28 grayscale digit is a point in $\mathbb{R}^{784}$, a token inside GPT-2 is a point in $\mathbb{R}^{768}$. Machine learning is geometry in spaces too big to draw, so every intuition you build in 2-D or 3-D is a tool you will reuse.

Vectors add component-wise, $\mathbf{x}+\mathbf{y} = (x_1+y_1,\dots,x_n+y_n)$, and scale component-wise, $c\mathbf{x} = (cx_1,\dots,cx_n)$. A linear combination $c_1\mathbf{v}_1 + \dots + c_k\mathbf{v}_k$ is the fundamental way to build new vectors from old ones; the set of all linear combinations is the span. If no vector in a set is a combination of the others, the set is linearly independent; an independent set that spans the space is a basis, and the number of basis vectors is the dimension. The standard basis $\mathbf{e}_1=(1,0,\dots)$, $\mathbf{e}_2=(0,1,\dots)$ is just one choice — embeddings live in a learned basis that is not aligned to any human-readable axis.

Intuition. A neural network layer outputs linear combinations of its inputs (then applies a nonlinearity). "Linear combination" is the verb of deep learning.

The dot product: alignment as a number

The dot product (inner product) of $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$ is

It has a geometric meaning that is the whole point:

where $\theta$ is the angle between the arrows. So the dot product is large and positive when the vectors point the same way, zero when they are orthogonal ($\theta = 90^\circ$), and negative when they oppose. Three consequences you will use constantly:

1. A neuron is a dot product. The pre-activation $z = \mathbf{w}^\top\mathbf{x} + b$ measures how well the input $\mathbf{x}$ aligns with the weight pattern $\mathbf{w}$; the weights are a template, the dot product is a match score (4.1 From perceptron to MLP; universal approximation).
2. Attention scores are dot products. In a transformer, how much token $i$ attends to token $j$ is $\mathbf{q}_i\cdot\mathbf{k}_j$ before softmax (7.2 Scaled dot-product and multi-head attention, fully derived).
3. Projection. The component of $\mathbf{x}$ along a unit vector $\mathbf{u}$ ($\|\mathbf{u}\|=1$) is the scalar $\mathbf{x}\cdot\mathbf{u}$, and the projected vector is $(\mathbf{x}\cdot\mathbf{u})\,\mathbf{u}$. For a general $\mathbf{v}$: $\operatorname{proj}_{\mathbf{v}}\mathbf{x} = \dfrac{\mathbf{x}\cdot\mathbf{v}}{\mathbf{v}\cdot\mathbf{v}}\,\mathbf{v}$. Least squares (3.2 Linear regression) and PCA (3.12 Dimensionality reduction: PCA, t-SNE and UMAP) are projection in disguise.

Cosine similarity removes length and keeps only direction:

This is how retrieval systems rank document chunks against a query (9.2 Embeddings deep dive): two texts "mean the same thing" when their embedding arrows point the same way, regardless of how long the texts are.

Norms: how big is a vector?

A norm assigns a length to a vector. The three you need:

They are all special cases of the $\ell_p$ norm $\|\mathbf{x}\|_p = \left(\sum_i |x_i|^p\right)^{1/p}$. The shape of the unit ball is not a curiosity: because the $\ell_1$ ball has corners on the axes, an optimizer constrained by it tends to land on solutions with exact zeros — that is why $\ell_1$ regularization produces sparse models (3.6 Regularization and generalization).

A norm gives a distance: $d(\mathbf{x}, \mathbf{y}) = \|\mathbf{x} - \mathbf{y}\|$. Euclidean distance is what kNN uses (3.4 k-Nearest Neighbors). Dividing a vector by its norm, $\hat{\mathbf{x}} = \mathbf{x}/\|\mathbf{x}\|_2$, puts it on the unit sphere; LayerNorm and RMSNorm (4.6 Initialization and normalization) are this idea applied to a layer's activations. Note the identity that connects distance and similarity for unit vectors: $\|\hat{\mathbf{x}}-\hat{\mathbf{y}}\|_2^2 = 2 - 2\cos\theta$, so for normalized embeddings "nearest by Euclidean distance" and "most similar by cosine" rank identically (9.2 Embeddings deep dive).

Worked example

Let $\mathbf{x} = (1, 2, 2)$ and $\mathbf{y} = (2, 0, 1)$.

• Dot product: $\mathbf{x}\cdot\mathbf{y} = 1\cdot 2 + 2\cdot 0 + 2\cdot 1 = 4$.
• Norms: $\|\mathbf{x}\|_2 = \sqrt{1+4+4} = 3$, $\;\|\mathbf{y}\|_2 = \sqrt{4+0+1} = \sqrt{5} \approx 2.236$.
• Angle: $\cos\theta = 4/(3\cdot 2.236) \approx 0.596$, so $\theta \approx 53.4^\circ$.
• Projection of $\mathbf{y}$ onto $\mathbf{x}$: $\dfrac{\mathbf{x}\cdot\mathbf{y}}{\mathbf{x}\cdot\mathbf{x}}\mathbf{x} = \dfrac{4}{9}(1,2,2) \approx (0.44, 0.89, 0.89)$.
• $\ell_1$ and $\ell_\infty$: $\|\mathbf{x}\|_1 = 5$, $\|\mathbf{x}\|_\infty = 2$.

Math check. The Cauchy–Schwarz inequality $|\mathbf{x}\cdot\mathbf{y}| \le \|\mathbf{x}\|\,\|\mathbf{y}\|$ is exactly the statement $|\cos\theta|\le 1$. Here $4 \le 6.708$. ✓

In NumPy:

import numpy as np
x, y = np.array([1., 2., 2.]), np.array([2., 0., 1.])
dot = x @ y                                  # 4.0
cos = dot / (np.linalg.norm(x) * np.linalg.norm(y))   # 0.596
proj = (x @ y) / (x @ x) * x                 # [0.444, 0.889, 0.889]

High dimensions are strange (and useful)

Pick two random vectors in $\mathbb{R}^n$ with independent standard-normal coordinates. Their cosine similarity has mean $0$ and standard deviation about $1/\sqrt{n}$: in $\mathbb{R}^{768}$ two random directions are almost always within a few degrees of orthogonal. Two consequences:

• Distances concentrate. Nearest and farthest neighbours become nearly equidistant, which hurts naive kNN in raw high-dimensional spaces (3.4 k-Nearest Neighbors) and is why we learn embeddings in which the geometry is meaningful.
• There is room for many "almost orthogonal" directions. A 768-dimensional space can hold far more than 768 nearly-orthogonal directions, which is one reason a model can pack many concepts into one residual stream — the phenomenon called superposition in interpretability (8.11 Interpretability: what is happening inside).

Where this shows up later

• Neurons compute $\mathbf{w}^\top\mathbf{x}+b$ — 4.1 From perceptron to MLP; universal approximation; matrices stack many such dot products — 1.2 Matrices as transformations.
• Euclidean distance drives kNN and k-means — 3.4 k-Nearest Neighbors, 3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical; norms define regularizers — 3.6 Regularization and generalization and gradient clipping — 4.5 Optimizers and learning-rate schedules.
• Dot products are attention scores — 7.2 Scaled dot-product and multi-head attention, fully derived; cosine similarity ranks retrieved chunks — 9.2 Embeddings deep dive; normalization layers rescale activations — 4.6 Initialization and normalization.