LearnAISign in
Phase 7 · 12 lessons · 36 h · about 4 weeks

⚡ Transformers

One architecture, derived line by line, from attention to frontier-scale inference

Lessons

7.1Attention Is All You Need: the big pictureWhy a 2017 translation paper became the blueprint for every frontier model2 h 30 min7.2Scaled dot-product and multi-head attention, fully derivedQ, K, V, the √d_k argument, masking and heads — every number computed by hand⚡ demo · 4 h7.3Positional information: sinusoidal, learned, RoPE, ALiBiAttention cannot see order — here is every way we give it one⚡ demo · 3 h7.4Inside the block: residuals, LayerNorm, FFN and embeddingsThe residual stream, the key–value memory, and how to count every parameter⚡ demo · 3 h7.5Encoder-only: BERT and friends (and sentence embeddings)Bidirectional encoders, masked language modelling, and where RAG's retrievers come from2 h 30 min7.6Decoder-only: GPT and autoregressive generationCausal LM objective, in-context learning, the generation loop and the Llama recipe2 h 30 min7.7Encoder–decoder: T5, BART and choosing an architectureSpan corruption, denoising, cross-attention — and a decision table for the three families2 h7.8Build GPT from scratch (nanoGPT walkthrough)From a bigram table to a Shakespeare generator — then to GPT-2 124M6 h7.9Scaling laws and compute-optimal trainingPower laws, FLOPs ≈ 6ND, Chinchilla's 20 tokens per parameter, and over-training⚡ demo · 2 h 30 min7.10Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantizationDecode is memory-bound — every trick here does more useful work per byte read⚡ demo · 3 h 30 min7.11Mixture of ExpertsRouting tokens to a few experts: huge capacity, small per-token compute2 h7.12Beyond vanilla attention: state space models, long context and hybridsSliding windows, linear attention, S4 and Mamba, and why 2026 models are hybrids2 h 30 min

About this phase

The transformer is the architecture behind every frontier language model, most vision and speech systems, and the agents you will build at the end of this path. This phase takes it apart completely. You derive scaled dot-product attention and compute it by hand, see why positions must be injected and how rotary embeddings do it, count the parameters of a block, and learn the three architectural families (encoder-only, decoder-only, encoder–decoder) and when each wins. Then you build a GPT from scratch, learn the scaling laws that decide how big a model should be, and understand what makes inference fast: KV caches, grouped-query attention, FlashAttention, speculative decoding and quantization. The phase closes with mixture-of-experts and the sub-quadratic alternatives (state space models, hybrids) that are reshaping the field in 2026.

By the end you can

  • Derive scaled dot-product and multi-head attention, including the √d_k scaling, and compute attention by hand for a small example
  • Explain sinusoidal, learned, rotary (RoPE) and ALiBi positional schemes and why they enable or limit length extrapolation
  • Count the parameters, FLOPs and KV-cache memory of any transformer configuration and apply Chinchilla-style compute-optimal reasoning
  • Choose between encoder-only, decoder-only and encoder–decoder designs for a task and justify the choice
  • Implement and train a small GPT, then read production code (nanoGPT, Llama) with full understanding
  • Describe the mechanics and trade-offs of MoE, FlashAttention, speculative decoding, quantization, and state-space/hybrid models