Multimodal models
One model, many senses: CLIP, vision-language models, text-to-image, speech and video
Frontier models now see, hear and draw. Under the hood a small set of ideas does almost all the work: a contrastive objective that aligns images and text in one space (CLIP, the InfoNCE of 5.8 Self-supervised and contrastive learning), a projector that turns image patches into tokens an LLM can read (5.7 Vision Transformers, 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings), cross-attention that lets a diffusion model follow a prompt (8.9 Generative models II: Diffusion and flow matching), and encoder–decoder transformers for speech (7.7 Encoder–decoder: T5, BART and choosing an architecture). Knowing them lets you read any multimodal model card, build multimodal RAG (9.7 Advanced RAG patterns), and give agents eyes (10.8 Coding agents and computer use).
You'll be able to
- Write CLIP's symmetric InfoNCE loss and explain zero-shot classification
- Describe how vision-language models connect a vision encoder to an LLM and what visual tokens cost
- Explain text conditioning in diffusion models via cross-attention and joint attention
- Summarize how Whisper and modern speech models process audio
- Identify evaluation benchmarks and typical failure modes of multimodal models
Key ideas
- CLIP aligns image and text encoders with a symmetric InfoNCE loss over an N×N similarity matrix; cosine similarity then enables zero-shot classification and retrieval.
- Vision-language models project vision-encoder patches into LLM token space (LLaVA recipe); images cost hundreds to thousands of tokens, and frontier models train natively multimodal.
- Text-to-image diffusion reads the prompt through cross-attention (U-Net) or joint attention (DiT), sharpened by classifier-free guidance.
Show 4 more key ideas
- Whisper is an encoder–decoder transformer over log-Mel spectrograms with task tokens; speech LLMs and codec-token generators extend the VLM recipe to audio.
- Video models diffuse spatiotemporal latents with transformer backbones — the same math at far higher compute.
- Early fusion (discrete tokens for every modality) vs late fusion (continuous features + separate generator) is the key architectural choice for unified models.
- Typical failures: object hallucination, counting, spatial reasoning, small text; evaluate with MMMU/MathVista/DocVQA and task-specific sets.
Suggested path through this lesson
One embedding space for images and text: CLIP
CLIP (2021) trains an image encoder (a ViT, 5.7 Vision Transformers) and a text encoder (a transformer) so that matching image–caption pairs have similar embeddings. For a batch of pairs, normalize the embeddings and form the similarity matrix with a learned temperature . The diagonal entries are the true pairs; everything else is a negative. The loss is a symmetric cross-entropy — InfoNCE (5.8 Self-supervised and contrastive learning) applied in both directions:
Trained on hundreds of millions of web pairs, the result is a space where cosine similarity (1.1 Vectors, dot products and norms) between an image and a sentence is meaningful. Zero-shot classification follows: embed prompts such as "a photo of a dog", "a photo of a cat", and pick the class whose text embedding is closest to the image. CLIP-style encoders (SigLIP replaces the softmax with a pairwise sigmoid loss) are the eyes of most vision-language models and the text encoders of many image generators; their embeddings also power multimodal retrieval (9.7 Advanced RAG patterns).
Intuition. Contrastive learning never says what a dog is; it only insists that the picture of a dog sits closer to the sentence about a dog than to 32,767 other sentences in the batch. Scale does the rest.
Vision-language models: image in, text out
The dominant recipe (LLaVA, 2023, and its many descendants) is modular:
- A pretrained vision encoder (CLIP/SigLIP ViT) turns an image into a grid of patch features.
- A projector — a linear layer or small MLP — maps them into the LLM's embedding space, producing visual tokens that are prepended to the text tokens.
- The LLM (7.6 Decoder-only: GPT and autoregressive generation) attends over visual and text tokens together and generates text.
Training proceeds in stages: align the projector on image–caption pairs with the encoder and LLM frozen, then instruction-tune on visual question–answer data, optionally unfreezing more. Earlier designs used learned resamplers (Flamingo's Perceiver, BLIP-2's Q-Former) to compress an image into a fixed number of query tokens. Frontier models are now trained natively multimodal, with interleaved image–text data from pretraining onward, but the token economics are the same: an image costs hundreds to a few thousand tokens depending on resolution and tiling, and video is many images. Resolution handling (tiling high-resolution inputs, dynamic token counts) is where much engineering effort goes.
Text-to-image: conditioning diffusion on language
A text encoder (CLIP's, or a large text-only encoder such as T5) produces token embeddings . In a U-Net backbone, cross-attention layers use image latents as queries and text tokens as keys and values (7.2 Scaled dot-product and multi-head attention, fully derived), so each spatial location can read the prompt. In transformer backbones (DiT, 8.9 Generative models II: Diffusion and flow matching), text and image tokens are concatenated and processed with joint attention. Classifier-free guidance (8.9 Generative models II: Diffusion and flow matching) sharpens prompt adherence. Prompt following, text rendering and counting improved mainly through better text encoders, better captions in training data, and scale.
Speech and audio
Whisper (2022) is an encoder–decoder transformer (7.7 Encoder–decoder: T5, BART and choosing an architecture): the encoder consumes a log-Mel spectrogram (audio as an image of frequencies over time) and the decoder emits text tokens, with special tokens that select the task (transcribe or translate), language and timestamps. It was trained on about 680,000 hours of weakly labeled web audio, which made it robust across accents and noise. Two later trends: speech LLMs that feed an audio encoder's outputs into an LLM through a projector (the VLM recipe applied to sound), and generation of speech via discrete audio tokens from neural codecs (EnCodec/SoundStream-style) predicted autoregressively, or via diffusion and flow models. Real-time speech-to-speech models skip text entirely, which preserves tone and allows interruptions.
Video, 3-D and the rest
Video generators extend latent diffusion/flow matching (8.9 Generative models II: Diffusion and flow matching) to spatiotemporal latents — a 3-D VAE compresses space and time, and a transformer attends over space–time patches. Compute is enormous and controllability is improving quickly; the same recipe handles audio-video jointly. Beyond media, the multimodal toolkit reaches proteins and molecules (sequence and structure models), robotics (vision-language-action models that output motor commands, 11.6 Frontier 2026: where the field is heading) and time series.
Tokenizing images and audio for a single autoregressive model
Two philosophies coexist. Late fusion keeps continuous visual features and a separate generator (the VLM + diffusion stack). Early fusion turns images and audio into discrete tokens with a VQ-VAE/VQGAN or codec (8.8 Generative models I: VAEs and GANs) so one autoregressive transformer reads and writes every modality in the same stream — the route behind unified models that both understand and generate images. Discrete tokens make training uniform; continuous latents keep more detail. Hybrids (autoregressive planning plus a diffusion decoder) are common.
Evaluation and failure modes
Benchmarks: MMMU (college-level multimodal questions), MathVista (visual math), DocVQA and ChartQA (documents and charts), TextVQA, video QA sets, and generation metrics such as FID and human preference for images. Typical failures: object hallucination (describing things not present), poor counting and spatial relations, misreading small text, biases in generated people and scenes, and brittleness to resolution. Safety adds deepfakes and provenance — watermarking and content credentials (C2PA) are the current, imperfect answers (8.12 Safety, risks and responsible AI).
Where this shows up later
- Multimodal RAG and document understanding — 9.7 Advanced RAG patterns; vision for computer-use agents — 10.8 Coding agents and computer use.
- Diffusion and flow backbones — 8.9 Generative models II: Diffusion and flow matching; contrastive learning — 5.8 Self-supervised and contrastive learning; ViT — 5.7 Vision Transformers.
- Frontier directions (world models, robotics) — 11.6 Frontier 2026: where the field is heading.
Common pitfalls (4)
- Feeding a VLM a tiny or heavily compressed image and blaming the model for missing small text.
- Ignoring image token costs when estimating latency and price for document pipelines.
- Expecting a CLIP similarity score to be calibrated — only rankings within a prompt set are meaningful.
- Assuming a model that describes images can also generate them (or vice versa); check the modality in/out of the model card.
Recap: all 7 key points
- CLIP aligns image and text encoders with a symmetric InfoNCE loss over an N×N similarity matrix; cosine similarity then enables zero-shot classification and retrieval.
- Vision-language models project vision-encoder patches into LLM token space (LLaVA recipe); images cost hundreds to thousands of tokens, and frontier models train natively multimodal.
- Text-to-image diffusion reads the prompt through cross-attention (U-Net) or joint attention (DiT), sharpened by classifier-free guidance.
- Whisper is an encoder–decoder transformer over log-Mel spectrograms with task tokens; speech LLMs and codec-token generators extend the VLM recipe to audio.
- Video models diffuse spatiotemporal latents with transformer backbones — the same math at far higher compute.
- Early fusion (discrete tokens for every modality) vs late fusion (continuous features + separate generator) is the key architectural choice for unified models.
- Typical failures: object hallucination, counting, spatial reasoning, small text; evaluate with MMMU/MathVista/DocVQA and task-specific sets.
Watch
Builds a PaliGemma-style VLM: SigLIP encoder, projector, LLM. Watch the architecture sections even if you skip the full code.
Read
Exercises
- CLIP zero-shot classifier
Load an open CLIP model, embed 10 images and 10 prompt templates ("a photo of a …"). Compute the cosine-similarity matrix and the zero-shot accuracy; then try bad prompts ("…") and see accuracy drop. Expected: prompt wording changes accuracy by several points.
- Symmetric InfoNCE in code
Implement the CLIP loss for a batch of paired embeddings in 10 lines of PyTorch (logits = normalized image @ text.T / τ; cross-entropy with labels arange(N) in both directions, averaged). Check that it equals at initialization with random embeddings. Expected: ≈ when the model is uninformative.
- Count visual tokens
For one open VLM, read its processor code or docs to find how many tokens a 1024×1024 image produces (tiles × tokens per tile). Estimate the cost of processing a 30-page scanned PDF. Expected: tens of thousands of tokens — the reason document pipelines downsample or crop.
- Whisper in practice
Transcribe a 2-minute noisy recording with an open Whisper model at two sizes; compare word error rate against your own transcript and note timestamps. Expected: the larger model is markedly more robust to noise and accents.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →