LearnAISign in
Phase 6 · 8 lessons · 25 h · about 3 weeks

✍ Sequences & NLP: the road to attention

From tokens and n-grams to recurrent networks and the first attention

Lessons

6.1Text to numbers: tokenizationHow language becomes a sequence of integers — and what gets lost⚡ demo · 2 h 30 min6.2Classical NLP and n-gram language modelsCounting words was the first language model — and it still earns its keep⚡ demo · 3 h6.3Word embeddingsMeaning as geometry: word2vec, GloVe and fastText3 h6.4Recurrent neural networks and BPTTSharing one set of weights across time — and paying for it in gradients3 h6.5LSTM and GRUGates: letting a network decide what to remember2 h 30 min6.6Seq2seq, encoder–decoder and beam searchReading a whole sentence into a vector, then writing a new one out2 h 30 min6.7Attention is bornA differentiable lookup that lets the decoder read the whole source2 h 30 min6.8Project: character-level language modelFrom bigram counts to an attention-equipped seq2seq, all from scratch6 h

About this phase

Language is the data type that produced the modern AI revolution, and this phase retraces the exact path the field walked to get there. You start by turning text into numbers — tokenization and the subword algorithms every LLM still uses — then learn the classical toolkit (TF-IDF, n-gram language models, perplexity) whose ideas survive inside RAG systems and evaluation. Word embeddings teach you that meaning can live in geometry. Recurrent networks, LSTMs and encoder–decoder models show how sequences were modelled before 2017, and why they struggled: the fixed-vector bottleneck and the sequential gradient path. The phase ends where the transformer begins — with attention, derived as a differentiable soft lookup. Every equation here is a prerequisite for Phase 7.

By the end you can

  • Explain and hand-simulate BPE tokenization and describe the failure modes it causes in LLMs
  • Compute TF-IDF vectors, n-gram probabilities with smoothing, and perplexity from a language model's log-probabilities
  • Derive the skip-gram negative-sampling objective and interpret embedding geometry (and its biases)
  • Write RNN, LSTM and GRU equations, derive backpropagation through time and explain vanishing/exploding gradients
  • Describe encoder–decoder models, teacher forcing and beam search, and derive Bahdanau attention as a soft dictionary lookup
  • Train a character-level language model and a tiny attention-based seq2seq model from scratch