Landmark papers

The first formal model of a neuron as a threshold logic unit, and a proof that networks of such units can compute any logical function. Every artificial neuron since is a descendant.

Replaced 'can machines think?' with the imitation game and argued that machines could learn like children. The conceptual starting point of the field.

The first learning algorithm for a neuron: adjust weights by the error. Its limitation to linearly separable problems set up the need for hidden layers.

Made backpropagation the standard way to train multilayer networks and showed hidden units learn useful internal representations.

The soft-margin SVM with the kernel trick: maximum-margin classification in implicit high-dimensional feature spaces, dominant for a decade.

Gated cells with an additive cell state that lets gradients survive long sequences; the workhorse of sequence modeling until the transformer.

LeNet-5: convolutional networks trained end to end with backprop for digit recognition, with the conv–pool–dense template still used today.

Bagged decision trees with random feature subsets at each split: a strong, low-tuning baseline that remains hard to beat on tabular data.

Framed boosting as gradient descent in function space, fitting each new tree to the loss gradient; the ancestor of XGBoost and LightGBM.

A nonlinear embedding that preserves local neighbourhoods, making high-dimensional data and embeddings visible in two dimensions.

AlexNet: GPUs, ReLU and dropout cut ImageNet top-5 error from 26% to 15% and started the deep learning era.

word2vec: dense word vectors learned by predicting context, with the famous analogy arithmetic (king − man + woman ≈ queen).

The variational autoencoder: the ELBO objective plus the reparameterization trick, making latent-variable models trainable with backprop.

DQN: a convolutional network as the Q-function with experience replay, learning to play Atari from pixels; deep RL begins.

Randomly dropping units during training prevents co-adaptation and acts like an ensemble; the default regularizer of the 2010s.

A generator and a discriminator in a minimax game; sharp samples without explicit likelihoods, and a decade of GAN research.

Encoder–decoder LSTMs for machine translation; established the seq2seq framing that attention and transformers inherited.

Attention is born: let the decoder look back at all encoder states with learned weights instead of one fixed vector.

Per-parameter adaptive steps from bias-corrected first and second moment estimates; the default optimizer for deep learning.

Word vectors from a weighted least-squares fit to co-occurrence statistics, unifying count-based and predictive embeddings.

Normalizing activations per mini-batch made deep networks train faster and with higher learning rates; the first mainstream normalization layer.

He initialization derived for ReLU networks plus PReLU; the first result above the human-level ImageNet baseline.

An encoder–decoder with skip connections for pixel-level prediction; still the backbone of segmentation and of early diffusion models.

Learned region proposals with anchors made two-stage detection end to end and fast enough to be practical.

Detection as a single regression over a grid: one network, one pass, real time. The YOLO lineage still dominates deployed detection.

Identity shortcuts (y = F(x) + x) made 100+ layer networks trainable; residual connections are now inside every transformer block.

AlphaGo combined policy and value networks with Monte Carlo tree search to beat a world champion, a decade earlier than expected.

Second-order gradient boosting with a regularized objective and systems optimizations; the engine behind years of Kaggle wins on tabular data.

HNSW: a layered proximity graph searched greedily; the most widely used index in vector databases.

Normalizing across features of one example rather than across the batch; works for sequences and became the transformer's normalization.

Conditional computation at scale: a trainable gate routes each input to a few experts, decoupling parameter count from per-example compute.

The transformer: self-attention replaces recurrence, enabling parallel training and constant path length between positions. The architecture of modern AI.

Learn a reward model from human comparisons, then optimize it with RL; the origin of RLHF.

A clipped surrogate objective that keeps policy updates small and stable; the RL algorithm behind classic RLHF.

AdamW: apply weight decay directly to weights instead of through the adaptive gradient; the optimizer used to train nearly every LLM.

GPT-1: pretrain a transformer decoder on next-token prediction, then fine-tune per task; the recipe that scaled into GPT-2 and GPT-3.

ELMo: word vectors that depend on context, from a bidirectional LSTM language model; the bridge from static embeddings to BERT.

Masked language modeling lets an encoder use both directions; fine-tuned BERT swept NLP benchmarks and seeded today's embedding models.

GPT-2: a 1.5B decoder trained on WebText performs tasks zero-shot; the first clear sign that scale buys generality.

Showed BERT was under-trained: more data, longer training and dropping next-sentence prediction mattered more than architecture.

Fine-tuning BERT in a siamese setup produces sentence vectors comparable with cosine similarity; the template for RAG embedding models.

T5: every NLP task as text-to-text with an encoder–decoder; a careful ablation of pretraining choices that shaped later recipes.

Loss follows smooth power laws in parameters, data and compute over many orders of magnitude; the empirical basis for betting on scale.

SimCLR: augmentations plus the InfoNCE loss learn visual features without labels; the same loss trains CLIP and text embedding models.

A dual-encoder trained with in-batch negatives beats BM25 for passage retrieval; dense retrieval becomes practical.

Late interaction: keep per-token embeddings and score with MaxSim, between bi-encoder speed and cross-encoder accuracy.

GPT-3 (175B) performs tasks from a few examples in the prompt: in-context learning improves smoothly with scale.

Couples a dense retriever with a generator, marginalizing over retrieved passages; the paper that named RAG.

A simple noise-prediction loss trains diffusion models that rival GANs in image quality; the start of the diffusion era.

A non-Markovian view of diffusion that allows deterministic sampling in far fewer steps.

The Vision Transformer: image patches as tokens; with enough pretraining data it matches or beats CNNs.

Unified diffusion and score matching as SDEs, giving a continuous-time theory and new samplers.

Top-1 routing with a load-balancing loss made mixture-of-experts transformers simple and stable at trillion-parameter scale.

CLIP: contrastive image–text pretraining on 400M pairs yields zero-shot classifiers and the conditioning signal for text-to-image models.

RoPE: rotate queries and keys by position so attention depends on relative distance; the positional encoding of Llama-style models.

Fine-tune by learning low-rank updates to frozen weights; trains a fraction of a percent of parameters with no inference cost.

Codex and the HumanEval benchmark with pass@k; established code generation as a core LLM capability and metric.

AlphaFold 2 solved single-chain protein structure prediction to experimental accuracy; recognized by the 2024 Nobel Prize in Chemistry.

Mask 75% of image patches and reconstruct them: BERT-style self-supervision that scales for vision.

Run diffusion in a pretrained autoencoder's latent space: Stable Diffusion, text-to-image on a consumer GPU.

Rewrote the transformer as a residual stream read and written by heads, enabling the circuit-level interpretability program.

Showing intermediate reasoning steps in the prompt unlocks multi-step reasoning in large models; the seed of reasoning models.

InstructGPT: SFT, a preference reward model and PPO make a 1.3B model preferred over 175B GPT-3; the recipe behind ChatGPT.

Chinchilla: for a fixed compute budget train on about 20 tokens per parameter; most earlier LLMs were badly under-trained.

Tiling attention to avoid materializing the score matrix in slow memory: exact attention, several times faster, enabling long contexts.

Steer diffusion toward a condition by extrapolating conditional and unconditional predictions; the knob behind every text-to-image model.

Networks represent more features than neurons by packing them into nearly orthogonal directions; explains polysemantic neurons.

Identified induction heads, a two-head circuit that copies earlier patterns, as a mechanism behind in-context learning.

Interleave thoughts, actions and observations; the trace format that became the standard tool-using agent loop.

Train a velocity field that transports noise to data along simple paths; a cleaner, faster alternative to diffusion that became the 2024–26 default.

Whisper: an encoder–decoder transformer trained on 680k hours of weakly labelled audio gives robust multilingual speech recognition.

Replace most human harmlessness labels with AI critiques and revisions guided by written principles; RLAIF at scale.

DiT replaces the U-Net with a transformer backbone and scales predictably; the architecture behind modern image and video generators.

A model learns where to insert API calls by self-supervision; an early demonstration of learned tool use.

Strong open-weight models trained on public data with RMSNorm, SwiGLU and RoPE; the recipe and the ecosystem of open LLMs.

A multimodal frontier model with predictable scaling of loss from small runs; also the point where labs stopped publishing architectures.

Agents improve by writing self-reflections on failures into memory instead of updating weights.

A promptable segmentation foundation model trained on a billion masks; segmentation as a zero-shot capability.

LLM agents with memory streams, reflection and planning produce believable social behaviour in a simulated town; the memory architecture influenced agent design.

LLaVA: connect a CLIP vision encoder to an LLM with a small projector and instruction-tune; the open recipe for vision-language models.

Search over branching reasoning states with self-evaluation; showed deliberate search beats a single chain on planning puzzles.

Share key/value heads across groups of query heads to shrink the KV cache with little quality loss; standard in modern LLMs.

LoRA on a 4-bit (NF4) quantized base model: fine-tune a 65B model on a single 48GB GPU.

Derives a closed-form loss on preference pairs from the RLHF objective, removing the reward model and the RL loop.

Measured how well strong LLMs grade other models and catalogued judge biases (position, verbosity, self-preference); the basis of arena-style evaluation.

Models use information at the beginning and end of a long context far better than in the middle; shaped how RAG orders evidence.

CoALA: a framework of memory, action space and decision procedure that organizes the zoo of agent designs.

Virtual-memory-style paging for the KV cache (vLLM) raised serving throughput several-fold; the default open serving stack.

Dictionary learning on activations recovers monosemantic features from superposition; the method behind large-scale feature maps of frontier models.

Real repository issues graded by hidden tests; the benchmark that defined progress in coding agents.

Treat the context window as RAM and external storage as disk, with the model paging memories in and out; the template for agent memory systems.

The model learns when to retrieve and critiques its own outputs with reflection tokens; a step toward agentic RAG.

Input-dependent state space models match transformers on language at small scale with linear-time inference; the strongest alternative architecture.

An open sparse MoE (8 experts, top-2) matching much larger dense models at a fraction of the active compute; made MoE mainstream in open weights.

Backdoored behaviours survived standard safety fine-tuning, showing current alignment methods can fail to remove hidden objectives.

Introduced GRPO, a critic-free policy optimization using group-normalized rewards; the algorithm behind DeepSeek-R1.

Ternary weights trained from scratch match full-precision quality at small scale, pointing to radically cheaper inference.

Build an entity graph and community summaries from a corpus so RAG can answer global 'what are the themes' questions.

Showed that the interface an agent is given (file viewer, search, edit commands) matters as much as the model; early strong SWE-bench results.

Simulated users plus policy rules plus tools, with the pass^k reliability metric that exposed how inconsistent agents are across trials.

A diffusion-based module predicts complexes of proteins, nucleic acids and ligands; generative modeling arrives in structural biology.

A detailed report on training 405B-parameter open-weight models, including data curation, scaling decisions and deliberate over-training for inference.

Distinguished workflows from agents and named the five composable patterns (chaining, routing, parallelization, orchestrator–workers, evaluator–optimizer) that practice converged on.

A model strategically complied during training it believed it was undergoing while behaving differently when unobserved; evidence that alignment must be verified, not assumed.

A 671B-parameter MoE with multi-head latent attention trained for a fraction of the usual cost; showed frontier-class open weights were achievable efficiently.

Reinforcement learning with verifiable rewards (GRPO) elicited long chains of thought and frontier-level reasoning in an open model; the reasoning-model recipe made public.

A thousand curated examples plus a budget-forcing trick reproduce much of test-time scaling; reasoning behaviour is cheaper to elicit than expected.

A taxonomy of failure modes in multi-agent systems (specification, coordination, verification) from hundreds of traces; a reality check on agent swarms.

Attribution graphs trace how a production model plans rhymes, does arithmetic and reasons across languages; interpretability at frontier scale.

Framed the agent's context as a curated, finite resource and described compaction, just-in-time retrieval and sub-agent patterns.

Practical guidance on grading agent trajectories, the pitfalls of nondeterminism and infrastructure noise, and building eval harnesses.

Decomposes agent memory into components and design decisions, consolidating the 2024–26 practice of memory beyond the context window.