Compute, hardware and scale basics

Why hardware shapes the field

Neural networks are mostly matrix multiplications, and matrix multiplication is embarrassingly parallel: every output element is an independent dot product (1.2 Matrices as transformations). GPUs pack thousands of simple cores plus tensor cores specialized for small matmuls; TPUs are systolic arrays built for nothing else; CPUs optimize single-thread latency and are 10–100× slower at this work. The architecture choices of the last decade — ReLU over sigmoid, transformers over RNNs, big batches — were all partly choices to fit the hardware. Today the binding constraint is less raw arithmetic than memory: how much fits, and how fast it moves.

The memory hierarchy

A GPU has a tiny amount of very fast on-chip memory (registers and SRAM, tens of megabytes at tens of terabytes per second), a large pool of high-bandwidth memory (HBM: 80 GB at roughly 3.35 TB/s on an H100), and beyond that the host's RAM and disk over a comparatively slow PCIe or NVLink bus. Moving a byte from HBM to the compute units costs far more time and energy than doing an arithmetic operation on it. Kernels that read and write HBM repeatedly (naive attention, element-wise chains) are memory-bound; kernels that do many operations per byte loaded (big matmuls) are compute-bound. FlashAttention (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization) is nothing but a rearrangement of attention to stay in SRAM; torch.compile's kernel fusion (4.9 PyTorch II: real training pipelines) is the same idea for element-wise ops.

Number formats

bf16's trade — fp32's range with less precision — is why it needs no loss scaling (4.8 Training dynamics and debugging) and why it won. Lower precision is faster and smaller: an H100 does roughly twice the FLOP/s in fp8 as in bf16, which is twice fp32-via-tf32.

Memory for training

Training a model with $N$ parameters in the standard mixed-precision recipe stores, per parameter: bf16 weights (2 bytes), bf16 gradients (2), an fp32 master copy (4) and Adam's two fp32 moments (8) — 16 bytes per parameter before anything else (4.5 Optimizers and learning-rate schedules). A 7-billion-parameter model therefore needs about $7\times10^{9}\times16 = 112$ GB just for its state: it does not fit on one 80 GB GPU, which is why full fine-tuning of 7B models requires either multiple GPUs or memory-saving methods (8.2 Fine-tuning and PEFT). Then come the activations saved for backward (4.4 Backpropagation, fully derived), which scale with batch size × sequence length × layers × hidden size (for a transformer layer roughly $34\,s\,b\,h$ bytes plus an attention term that FlashAttention removes), and are usually the largest term for long sequences. Activation checkpointing recomputes them during backward to trade ~30% compute for memory, and gradient accumulation (4.9 PyTorch II: real training pipelines) reduces the live batch. Finally there are temporary buffers and fragmentation; budget 10–20% headroom.

Memory for inference

Inference needs weights plus a KV cache (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization) — no gradients, no optimizer. Weights in bf16 cost $2N$ bytes: 14 GB for a 7B model, 140 GB for a 70B model; int8 halves that and int4 quarters it (a 7B model fits in ~4 GB), which is how 7B–8B models run on laptops and 70B models on two consumer GPUs. The KV cache adds $2\times L\times n_{\text{kv}}\times d_{\text{head}}\times\text{seq}\times\text{bytes}$ per sequence and often dominates at long contexts and high batch sizes.

FLOPs: the 6ND rule

A matmul of an $m\times k$ by a $k\times n$ matrix costs $2mkn$ FLOPs (one multiply and one add per term). Pushing one token through a layer with weight matrices totalling $N_\ell$ parameters costs $\approx 2N_\ell$ FLOPs, so a forward pass costs about $2N$ FLOPs per token for a model with $N$ (non-embedding) parameters — attention's $n^2$ terms are a small correction unless the context is very long. The backward pass costs about twice the forward (4.4 Backpropagation, fully derived), so training costs $\approx 6N$ FLOPs per token, and a run over $D$ tokens costs

the formula at the heart of scaling laws (7.9 Scaling laws and compute-optimal training). Real hardware never hits its peak; model FLOPs utilization (MFU) — achieved useful FLOP/s divided by peak — is typically 30–50% for well-tuned large runs.

Worked example 1. GPT-2 small ($N=124$M) on $D=10$B tokens: $C = 6\times1.24\times10^{8}\times10^{10} = 7.4\times10^{18}$ FLOPs. An H100 peaks near $9.9\times10^{14}$ bf16 FLOP/s; at 40% MFU that is $4\times10^{14}$, so the run takes $7.4\times10^{18}/4\times10^{14}\approx1.9\times10^{4}$ s $\approx$ 5 hours. Karpathy's live reproduction (7.8 Build GPT from scratch (nanoGPT walkthrough)) lands in this ballpark.

Worked example 2. Llama 3 405B on 15.6T tokens: $C\approx6\times4.05\times10^{11}\times1.56\times10^{13}\approx3.8\times10^{25}$ FLOPs. On 16,384 H100s at 40% MFU ($6.5\times10^{18}$ FLOP/s) that is $5.8\times10^{6}$ s $\approx$ two months — consistent with the reported training time, and the reason frontier training is a capital project.

Arithmetic intensity and the roofline

Arithmetic intensity is FLOPs performed per byte moved from HBM. The hardware has a ridge point — peak FLOP/s divided by bandwidth, about $9.9\times10^{14}/3.35\times10^{12}\approx300$ FLOP/byte for an H100 in bf16 — below which a kernel is memory-bound and above which it is compute-bound. A weight matrix multiplied by a batch of $b$ token vectors has intensity roughly $b$ (each weight byte loaded is used for $b$ tokens), so training with thousands of tokens per batch is compute-bound, but decoding one token at a time for one user is hopelessly memory-bound: the GPU reads every weight to produce one token, and tokens-per-second is capped by bandwidth divided by model bytes — about $3.35\times10^{12}/1.4\times10^{10}\approx240$ tokens/s for a 7B bf16 model, regardless of FLOPs. Batching requests, quantization and speculative decoding (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization) are all attacks on this limit.

Parallelism: using more than one accelerator

• Data parallelism (DDP, 4.9 PyTorch II: real training pipelines): replicate the model, split the batch, all-reduce the gradients. Communication per step is proportional to $N$; works until the model state no longer fits on one device.
• ZeRO / FSDP: shard the optimizer states, gradients and finally the parameters across the $W$ data-parallel workers, so per-device memory falls from 16 bytes per parameter toward $16/W$; parameters are gathered just-in-time per layer. This is how most 7B–70B fine-tuning and pretraining is done on a single node.
• Tensor (model) parallelism: split individual weight matrices across devices (each computes a slice of the matmul) and exchange partial results every layer — needs very fast interconnect (NVLink), so it stays within a node.
• Pipeline parallelism: assign groups of layers to different devices and stream micro-batches through them; "bubbles" of idle time are reduced by interleaving schedules.
• Expert parallelism places different MoE experts on different devices (7.11 Mixture of Experts); sequence/context parallelism splits very long sequences (ring attention, 7.12 Beyond vanilla attention: state space models, long context and hybrids).

Frontier runs combine all of these ("4D parallelism"), and the Hugging Face Ultra-Scale Playbook walks through the choices with real measurements. For a learner, the practical ladder is: one GPU → gradient accumulation → DDP → FSDP.

Throughput, latency and cost

Training cares about throughput (tokens/s, hence big batches); serving cares about latency (time to first token, tokens per second per user) and cost per token. To budget a run: FLOPs $\div$ (peak $\times$ MFU) gives seconds; multiply by the GPU-hour price (cloud H100s have hovered around $2–4 per hour in 2025–26; consumer cards like an RTX 4090 with 24 GB cost about as much as a few hundred cloud hours). Worked example 1 above costs under $20 — reproducing GPT-2 is a weekend project, not a lab budget.

Choosing hardware as a learner

Free Colab and Kaggle GPUs (T4/P100 with 16 GB, occasionally better) cover Phases 4–7 comfortably: every MLP, CNN, char-RNN and small GPT in this path trains on them. A 12–24 GB consumer GPU runs 7B–8B models in 4-bit for inference and QLoRA fine-tuning (8.2 Fine-tuning and PEFT); Apple-silicon unified memory is excellent for inference and slow for training. Rent an H100 by the hour for anything larger, and benchmark MFU before scaling a run. The rule: estimate first, buy never (until you must). The llm-calculator demo below does the arithmetic for parameter counts, memory, FLOPs and training time.

Where this shows up later

• Transformer parameter counting and the memory of the KV cache, GQA and quantization — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings, 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization; the 6ND rule and compute-optimal training — 7.9 Scaling laws and compute-optimal training; MoE's active-vs-total parameters — 7.11 Mixture of Experts.
• Why QLoRA fits a 7B fine-tune in 10 GB and full fine-tuning does not — 8.2 Fine-tuning and PEFT; serving costs, batching and caching in production — 9.9 Production: latency, cost, caching, guardrails, security and privacy; agent cost control, where tokens are money — 10.11 Building production agents.
• Mixed-precision debugging — 4.8 Training dynamics and debugging; DDP/FSDP in the training template — 4.9 PyTorch II: real training pipelines.