Project: fine-tune a small LLM with LoRA and ship a chat app

The goal and the deliverables

Pick one narrow, demonstrable task where fine-tuning plausibly beats prompting (8.2 Fine-tuning and PEFT): a consistent persona for a support bot, structured extraction from messy text into JSON, a domain-specific classification-plus-explanation, or a stylistic rewriting task. Avoid "teach it facts" — that is retrieval (9.11 Project: a cited RAG assistant over your own documents, with evals). Deliverables: a GitHub repository with the data pipeline, training config, evaluation script and results table; a merged (and quantized) model on the Hugging Face Hub; a deployed Gradio or Streamlit chat app; and a two-page write-up.

Step 1 — choose the model (1 hour)

Use a 1–8B instruct model from a permissively licensed family (small Qwen, Llama, Gemma, Mistral or Phi variants — check each license for your use). Instruct bases need less data than raw bases for assistant-style tasks. Confirm the chat template, context length and tokenizer behavior on your domain's text (6.1 Text to numbers: tokenization). On a free Colab/Kaggle GPU (16 GB), QLoRA on a 1–4B model trains comfortably; 7–8B works with short sequences and small batches.

Step 2 — build the dataset (3–4 hours)

Target 500–3,000 examples. Sources: your own writing, public datasets filtered to your task, or synthetic generation by a strong model from seed examples (check its terms), followed by filtering with rules and spot checks. Clean aggressively: deduplicate, remove examples that violate your format, fix inconsistent labels, and read a random sample of 50 by hand. Render every example through the chat template, mask prompt tokens from the loss, and split into train/validation/test by source when possible to avoid leakage. Write a 20–50 item golden test set with expected outputs or rubrics before training (8.7 Evaluating LLMs).

Step 3 — train with QLoRA (1–2 hours of compute)

Use TRL's SFT trainer with PEFT (or Unsloth for speed). A reasonable starting configuration: 4-bit NF4 base, LoRA $r=16$, $\alpha=32$, dropout 0.05, all attention and MLP projections targeted, learning rate $2\times10^{-4}$ with cosine decay and a short warmup, effective batch size 16 via gradient accumulation, sequence length 1,024–2,048, 1–3 epochs. Log training and validation loss every few dozen steps; stop when validation loss plateaus. Watch GPU memory and tokens per second so you can explain your choices.

from trl import SFTConfig, SFTTrainer
from peft import LoraConfig

peft_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                         target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"])
args = SFTConfig(output_dir="out", num_train_epochs=2, learning_rate=2e-4, lr_scheduler_type="cosine",
                 warmup_ratio=0.03, per_device_train_batch_size=4, gradient_accumulation_steps=4,
                 max_length=1024, logging_steps=20, eval_strategy="steps", eval_steps=100, bf16=True)
trainer = SFTTrainer(model=model, args=args, train_dataset=train_ds, eval_dataset=val_ds,
                     processing_class=tokenizer, peft_config=peft_config)
trainer.train()

Argument names drift between library versions; read the current docs rather than copying blindly.

Step 4 — evaluate honestly (2 hours)

Compare three systems on the golden set: (a) the base instruct model with a carefully written prompt and 3 few-shot examples, (b) your fine-tune with a short prompt, (c) optionally a frontier API model as a ceiling. Score with deterministic checks where possible (schema validity, exact fields, classification accuracy) and with an LLM judge for open-ended quality — pairwise, both orders, a rubric, a different model family as judge (8.7 Evaluating LLMs). Also run a small regression check (20 general questions) to detect catastrophic forgetting (8.2 Fine-tuning and PEFT). Report confidence intervals; with 50 items, differences under ~10 points are noise. Fix decoding settings for all systems (8.5 Decoding and sampling).

Step 5 — merge, quantize, serve (1–2 hours)

Merge the adapter into the base weights, push to the Hub with a model card (data, training config, evals, limitations, license). For local serving, export to GGUF with llama.cpp and run under Ollama, or serve the merged model with vLLM (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization) for throughput. Measure latency and tokens per second at your target quantization and confirm the eval scores hold after quantization.

Step 6 — ship the chat app (1–2 hours)

Build a Gradio chat interface (or Streamlit) with streaming tokens, a system prompt, and sampling controls exposed as sliders. Add a visible disclaimer and a feedback button that logs thumbs up/down with the conversation (with consent). Deploy to Hugging Face Spaces or a small VM. Add basic guardrails from 8.12 Safety, risks and responsible AI: input length limits, a refusal path for out-of-scope requests, no secrets in the prompt.

Step 7 — write it up

Two pages: the task and why fine-tuning was the right tool; dataset construction with statistics; training curves and configuration; the results table with intervals and three qualitative examples (including a failure); cost and latency; limitations and what you would do next. Portfolio reviewers reward honesty and measurement over hype (11.3 Portfolio project ladder).

Rubric

Stretch goals

Collect 200 preference pairs from your judge or users and run a DPO stage (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI); distill reasoning traces for a harder variant (8.6 Reasoning models and test-time compute); add retrieval for the factual parts (9.11 Project: a cited RAG assistant over your own documents, with evals); or compare three ranks and two base sizes and plot quality versus cost.

Where this shows up later

• The RAG capstone — 9.11 Project: a cited RAG assistant over your own documents, with evals — and the agent capstone — 10.12 Project: build a research agent with tools, an MCP server and an eval harness — reuse this evaluation discipline.
• Portfolio and interview framing — 11.3 Portfolio project ladder, 11.4 ML and LLM system design and interviews.