Fine-tuning and PEFT
Teach a model new behavior with a fraction of the parameters
Fine-tuning is how a general model becomes your model — a consistent style, a domain vocabulary, a structured output format, or a small cheap model that imitates a big one. Full fine-tuning of billions of parameters is out of reach for most people, so parameter-efficient methods, above all LoRA, carry the field. The low-rank idea comes straight from 1.3 Eigen-things, SVD and low rank; the training mechanics from 4.5 Optimizers and learning-rate schedules; and the decision of whether to fine-tune at all depends on what retrieval (9.1 Why RAG, and the canonical pipeline) and prompting (8.4 Prompting as programming) can already do.
You'll be able to
- Decide between prompting, retrieval and fine-tuning for a concrete requirement
- Prepare an SFT dataset with the correct chat template and loss masking
- Derive LoRA's parameter savings and implement it with Hugging Face PEFT
- Explain QLoRA's 4-bit NormalFloat quantization and why adapters stay in higher precision
- Detect and mitigate overfitting and catastrophic forgetting in a fine-tune
Key ideas
- Retrieve for facts, fine-tune for form: fine-tuning changes behavior, style and skills, not up-to-date knowledge.
- SFT success depends on the chat template, loss masking on assistant tokens, and a small set of high-quality examples.
- Full fine-tuning costs ~16 bytes per parameter with Adam in mixed precision, so a 7B model needs ~112 GB before activations.
Show 4 more key ideas
- LoRA learns ΔW = (α/r)·BA with B zero-initialized; r(d+k) parameters instead of dk — e.g. 0.39% for a 4096² matrix at r = 8.
- Merging W₀ + (α/r)BA restores the original architecture, so LoRA adds no inference latency.
- QLoRA stores frozen weights in 4-bit NormalFloat with double quantization and keeps adapters in higher precision — a 65B fine-tune fits on one 48 GB GPU.
- Always evaluate the fine-tune against a well-prompted baseline on a held-out set, and watch for catastrophic forgetting.
Suggested path through this lesson
Should you fine-tune at all?
Fine-tuning changes the model's weights; prompting changes its input; retrieval changes its context. They solve different problems:
| you need… | first try | why |
|---|---|---|
| up-to-date or private facts | retrieval (RAG, 9.1 Why RAG, and the canonical pipeline) | facts change; weights are a poor database; citations possible |
| a particular format, tone or persona | prompting, then fine-tuning | few-shot examples often suffice; fine-tune when prompts get long or inconsistent |
| a skill the model lacks (a DSL, a classification scheme, tool protocol) | fine-tuning | behavior that is hard to describe but easy to demonstrate |
| lower cost/latency at equal quality | fine-tune a small model on a big model's outputs (distillation) | pay once in training, save on every call |
| strict reliability on a narrow task | fine-tuning + constrained decoding (8.5 Decoding and sampling) | shrinks the output distribution |
The rule of thumb: retrieve for facts, fine-tune for form. Always build the evaluation set first (8.7 Evaluating LLMs) — you need it to know whether fine-tuning helped and whether it hurt anything else.
Supervised fine-tuning data
SFT data is a set of conversations: a system prompt (optional), user turns and ideal assistant turns. Three mechanics matter:
- The chat template. Each model family serializes conversations with its own special tokens. Use
tokenizer.apply_chat_template; a mismatch is the single most common fine-tuning bug. - Loss masking. Compute the next-token loss only on assistant tokens (set labels to for prompt tokens in PyTorch), so the model learns to answer, not to imitate your prompts.
- Quality over quantity. LIMA showed that about 1,000 carefully written examples can produce a strong assistant from a good base model; noisy data teaches noise. Deduplicate, decontaminate against your eval set, and inspect a random sample by hand.
Data can be human-written, filtered from production logs (with consent), or synthetic — generated by a stronger model (Self-Instruct style), then filtered. Check the stronger model's terms of service before distilling from it.
Why full fine-tuning is expensive
Training a parameter with Adam in mixed precision costs far more memory than storing it: roughly 2 bytes for the bf16 weight, 2 for its gradient, and 12 for the fp32 master copy plus the two Adam moments — about 16 bytes per parameter before activations. A 7B model therefore needs about 112 GB just for weights and optimizer state: several GPUs for a job that may only change the model slightly. Hence parameter-efficient fine-tuning (PEFT).
LoRA: low-rank adaptation
Hypothesis. The change a fine-tune makes to a weight matrix has low intrinsic rank — most of the update lives in a few directions (recall the SVD view of low-rank structure from 1.3 Eigen-things, SVD and low rank).
Construction. Freeze the pretrained matrix and learn a low-rank update:
is initialized with small random values and with zeros, so at the start and training begins exactly at the pretrained model. The scale keeps the update magnitude roughly constant when you change ; is a common default.
Parameter count. The update has trainable numbers instead of .
Math check. Take a attention projection and . Full: parameters. LoRA: — about 0.39%, a 256× reduction. Applying LoRA to all attention and MLP projections of a 7B model with yields on the order of 20M trainable parameters, roughly 0.3% of the model, and optimizer memory shrinks accordingly.
Inference. After training, merge: . The merged model has exactly the original architecture and zero extra latency. Keep adapters unmerged when you want to swap several task-specific adapters over one base model.
Hyperparameters that matter. Rank (8–64; higher is not always better), which modules to target (attention projections are the minimum; adding the MLP layers usually helps), learning rate (higher than full fine-tuning, typically to ), dropout on the adapter, and 1–3 epochs.
from peft import LoraConfig, get_peft_model, TaskType
config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters() # e.g. 0.6% of total
QLoRA: fine-tune a 65B model on one GPU
QLoRA keeps the frozen base weights in 4-bit NormalFloat (NF4), a data type whose 16 levels are placed at the quantiles of a standard normal distribution — the right choice because trained weights are approximately normally distributed, so each level is used equally often. Blocks of weights share a scale factor, and those scales are themselves quantized (double quantization) to save a further ~0.4 bits per parameter. During the forward pass each block is dequantized to bf16 on the fly, multiplied, and discarded; the LoRA adapters and their optimizer states stay in bf16/fp32. Paged optimizers move optimizer state to CPU memory when the GPU spikes. The result: a 65B model fine-tuned on a single 48 GB GPU with quality matching 16-bit LoRA, at the price of ~30% slower steps.
Other PEFT methods, briefly
| method | what is trained | notes |
|---|---|---|
| adapters | small bottleneck MLPs inserted in each block | adds inference latency unless merged |
| prefix tuning | learned key/value prefixes per layer | strong on generation, awkward to tune |
| prompt tuning | learned soft tokens in the embedding space | matches fine-tuning only at large scale |
| (IA)³ / DoRA | per-channel rescaling / magnitude-direction split of LoRA | small gains over LoRA in some settings |
| LoRA / QLoRA | low-rank on chosen matrices | the default in 2026 |
Failure modes
- Catastrophic forgetting: general ability drops after narrow fine-tuning. Mitigate by mixing in general instruction data, lowering the learning rate, fewer epochs, and smaller ; LoRA itself is a strong regularizer.
- Overfitting the format: validation loss keeps falling while judged quality falls — the model memorizes surface patterns. Evaluate with a held-out set and a judge (8.7 Evaluating LLMs).
- Template or masking bugs: the fine-tune "works" in training logs but the deployed model rambles or echoes prompts.
- Evaluation against the wrong baseline: compare with the base instruct model under a good prompt, not with no prompt at all.
Tooling in 2026
Hugging Face PEFT (adapters) and TRL (SFT, DPO, GRPO trainers) are the reference stack; Unsloth gives large speed and memory gains for single-GPU QLoRA; Axolotl and LLaMA-Factory are configuration-driven alternatives; and API providers offer hosted fine-tuning when you cannot host weights. Free Colab/Kaggle GPUs are enough for QLoRA on 1–8B models — exactly what the capstone 8.13 Project: fine-tune a small LLM with LoRA and ship a chat app does.
Where this shows up later
- Preference optimization (DPO) is a fine-tune with a different loss and usually LoRA adapters — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI.
- Reasoning distillation fine-tunes small models on long chains of thought — 8.6 Reasoning models and test-time compute.
- When fine-tuning is the wrong tool, retrieval is the right one — 9.1 Why RAG, and the canonical pipeline; evaluating either needs 8.7 Evaluating LLMs.
Common pitfalls (5)
- Fine-tuning to inject facts: the model will state them inconsistently and still hallucinate; use retrieval.
- Training on prompt tokens too (no loss masking), which teaches the model to generate user questions.
- Using a different chat template at inference than at training time.
- Comparing the fine-tuned model to an unprompted base model and declaring victory.
- Pushing r to 256 'to be safe' — more adapter capacity mostly buys more overfitting on small datasets.
Recap: all 7 key points
- Retrieve for facts, fine-tune for form: fine-tuning changes behavior, style and skills, not up-to-date knowledge.
- SFT success depends on the chat template, loss masking on assistant tokens, and a small set of high-quality examples.
- Full fine-tuning costs ~16 bytes per parameter with Adam in mixed precision, so a 7B model needs ~112 GB before activations.
- LoRA learns ΔW = (α/r)·BA with B zero-initialized; r(d+k) parameters instead of dk — e.g. 0.39% for a 4096² matrix at r = 8.
- Merging W₀ + (α/r)BA restores the original architecture, so LoRA adds no inference latency.
- QLoRA stores frozen weights in 4-bit NormalFloat with double quantization and keeps adapters in higher precision — a 65B fine-tune fits on one 48 GB GPU.
- Always evaluate the fine-tune against a well-prompted baseline on a held-out set, and watch for catastrophic forgetting.
Watch
The low-rank hypothesis, the math and a from-scratch implementation.
Read
Try it live
Interactive demo
runs in your browserExercises
- LoRA by hand
For a model with 32 layers, each with four attention projections and MLP matrices of (two up, one down), count full parameters versus LoRA parameters at applied to all of them. Expected: well under 1% at ; note how the MLP matrices dominate the count.
- Implement LoRA in 30 lines
Write a
LoRALinearPyTorch module that wraps a frozennn.Linearand adds with zero-initialized. Verify that the output equals the frozen layer at initialization and that only receive gradients. Then writemerge()and check the merged weight reproduces the outputs. - Chat-template forensics
Take a small instruct model and print
tokenizer.apply_chat_template(messages, tokenize=False). Identify every special token. Then deliberately fine-tune 50 steps with a wrong template and observe the behavior change — a lesson you will not forget. - Decision memo
For three product requests (brand voice for marketing copy; answering questions about this month's policy PDF; extracting invoices into JSON), write one paragraph each choosing prompting, RAG or fine-tuning, with the eval you would build first.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →