Landmark papers: the reading list

How to use this list and the timeline page

The Papers page in this site (the timeline link in the navigation) lists about seventy landmark papers from 1943 to 2026, each with the lesson that discusses it and a one-line "why". Filter it by tag when you finish a phase: after Phase 5 read the vision entries, after Phase 7 the transformer entries, and so on. The ten below are the spine. For each, do pass one and pass two from 11.1 How to read (and reproduce) papers; do pass three for the two or three closest to your interests.

The ten papers

1. Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014). Read it for the update rule and the bias-correction argument: with $m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t$ and $v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2$, the corrected estimates $\hat m_t=m_t/(1-\beta_1^t)$ and $\hat v_t=v_t/(1-\beta_2^t)$ give the step $\theta_t=\theta_{t-1}-\eta\,\hat m_t/(\sqrt{\hat v_t}+\epsilon)$. Notice how little tuning the paper claims is needed, and then read Decoupled Weight Decay Regularization (AdamW, 2017), which fixed the way weight decay interacts with the adaptive denominator and is what everyone actually uses (4.5 Optimizers and learning-rate schedules).

2. Deep Residual Learning for Image Recognition (He et al., 2015). The one-line idea $\mathbf{y}=F(\mathbf{x})+\mathbf{x}$ made 100+ layer networks trainable and is now inside every transformer block. Pay attention to the degradation experiment (deeper plain nets getting worse training error, so the problem was optimization, not overfitting) and to the bottleneck design. Follow-ups: pre-activation ResNets, and the residual-stream view of transformers in 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings.

3. Attention Is All You Need (Vaswani et al., 2017). Read Section 3 slowly: $\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}(\mathbf{QK}^\top/\sqrt{d_k})\mathbf{V}$, multi-head attention, sinusoidal positions, and Table 1 comparing per-layer complexity and path length, which is the real argument for dropping recurrence. Then notice what the paper does not have: no decoder-only variant, no pre-norm, no RoPE. Those arrived in GPT, GPT-2 and later Llama-style recipes (7.6 Decoder-only: GPT and autoregressive generation).

4. BERT (Devlin et al., 2018). The contribution is a training objective, masked language modeling, that lets an encoder see both directions and transfer to many tasks with one extra layer. Study the input representation (token + segment + position), the 15 percent masking rule and the fine-tuning results. Then read RoBERTa (what mattered was data and training time, not next-sentence prediction) and Sentence-BERT, which turned encoders into the embedding models behind RAG (7.5 Encoder-only: BERT and friends (and sentence embeddings), 9.2 Embeddings deep dive).

5. Language Models are Few-Shot Learners (GPT-3, Brown et al., 2020). The paper is long; pass two is enough for most readers. The idea to extract is in-context learning: a 175B-parameter decoder trained only on next-token prediction performs tasks from a few examples in the prompt, with performance improving smoothly with scale (Figure 1.2). Read Section 5 on limitations and Section 6 on broader impacts, which anticipated much of 2022–2025. Follow-up: InstructGPT, below (7.6 Decoder-only: GPT and autoregressive generation).

6. Scaling Laws for Neural Language Models (Kaplan et al., 2020). The claim is that loss follows power laws in parameters $N$, data $D$ and compute $C$, for example $L(N)\approx (N_c/N)^{\alpha_N}$. Learn to read the log-log plots. Then read Training Compute-Optimal Large Language Models (Chinchilla, 2022), which corrected the recommended allocation to roughly twenty tokens per parameter, and note how Llama 3 deliberately over-trained for inference efficiency (7.9 Scaling laws and compute-optimal training).

7. Training language models to follow instructions with human feedback (InstructGPT, Ouyang et al., 2022). This is the recipe behind ChatGPT: supervised fine-tuning, a reward model trained on pairwise human preferences with the Bradley–Terry loss, and PPO against that reward with a KL penalty to the SFT model. Figure 2 is the whole pipeline. Follow with Direct Preference Optimization (2023), which removes the reward model and the RL loop, and DeepSeek-R1 (2025) for RL with verifiable rewards (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI, 8.6 Reasoning models and test-time compute).

8. Denoising Diffusion Probabilistic Models (Ho et al., 2020). The forward process adds Gaussian noise with a schedule; the model learns to predict the noise; the simplified loss $\mathbb{E}\left[\|\epsilon-\epsilon_\theta(\mathbf{x}_t,t)\|^2\right]$ works better than the full variational bound. Work through Section 3 with the unified-perspective tutorial by Calvin Luo beside it. Follow-ups: DDIM for fast sampling, classifier-free guidance, latent diffusion, and flow matching, which is the 2024–26 default (8.9 Generative models II: Diffusion and flow matching).

9. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020). The original RAG trained a retriever and a generator jointly, marginalizing over retrieved passages. Modern RAG kept the idea and dropped the joint training: retrieve with an embedding model, stuff the context, generate. Read it to understand what "RAG" originally meant and why the engineering practice in 9.1 Why RAG, and the canonical pipeline diverged from it; then read the 2024 survey and Self-RAG.

10. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022). The paper interleaves "thought" text with actions and observations, and shows that reasoning helps acting and acting grounds reasoning. The trace format in Figure 1 is the ancestor of every tool-calling agent loop (10.2 Tool use and function calling, 10.3 Reasoning and planning for agents). Follow with Toolformer, Reflexion and Anthropic's Building effective agents for what practice converged on.

Reading order by phase

Where this shows up later

• The project ladder in 11.3 Portfolio project ladder ends with a paper reproduction; pick it from this list or the timeline page.
• System-design interviews in 11.4 ML and LLM system design and interviews routinely ask "why transformers over RNNs" or "RLHF vs DPO": the answers are in papers 3 and 7.
• The frontier in 11.6 Frontier 2026: where the field is heading is written in the language of these ten papers.