ML and LLM system design and interviews
A framework, two worked cases and the question families you will meet
Most ML and AI engineering roles are won in two rounds: a system-design conversation where you reason from requirements to monitoring, and technical rounds that probe fundamentals (3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 4.4 Backpropagation, fully derived, 7.2 Scaled dot-product and multi-head attention, fully derived) and LLM engineering (9.9 Production: latency, cost, caching, guardrails, security and privacy, 10.10 Agent safety and security). The same framework you use to pass those rounds is the one you should use to design real systems, so this lesson teaches both at once.
You'll be able to
- Apply a six-step framework (requirements → data → model → serving → evaluation → monitoring) to any ML or LLM product
- Design a two-stage recommendation or search system and an LLM assistant with RAG, with explicit trade-offs
- Recognize the five interview question families and prepare for each with a plan
- Communicate designs with clarifying questions, numbers and stated assumptions
Key ideas
- Six steps, said aloud: requirements → data → model → serving → evaluation → monitoring.
- Turn goals into numbers (latency percentiles, cost per request, target metric) before proposing a model.
- Recommendation and search use two stages: cheap candidate generation (two-tower embeddings + ANN) and a richer ranker; evaluate with recall@k, nDCG and time-split A/B tests.
Show 4 more key ideas
- An LLM assistant with RAG needs hybrid retrieval + reranking, a cited grounded prompt with a refusal path, tool calls for actions, and access control enforced in the store.
- Evaluate LLM systems with a golden set, a judge calibrated against humans and an adversarial injection set; monitor refusal rate, judge scores and cost.
- Five question families: fundamentals, coding, ML system design, LLM engineering, behavioral; prepare each from your own projects.
- Always start from a baseline and say what evidence would justify each upgrade; finish with evaluation and monitoring.
Suggested path through this lesson
The framework
Every design conversation, whether in an interview or a planning meeting, can follow six steps. Say them out loud; interviewers reward structure.
- Requirements. What is the business goal and the user-facing behaviour? What are the constraints: latency budget, request volume, cost per request, privacy, regulation, offline vs online? Ask clarifying questions before proposing anything; a candidate who designs the wrong system beautifully still fails. Turn goals into measurable targets: "p95 latency under 2 s, answer relevance above 0.85, cost under $0.02 per query".
- Data. Where do labels come from (explicit, implicit, synthetic, human review)? How will you split train, validation and test without leakage (3.5 Evaluating models: metrics, validation and the bias–variance trade-off), especially in time? How large is the data, how fast does it drift, what is the cold-start situation, what is sensitive?
- Model. Start with the simplest baseline that could work (a heuristic, logistic regression, BM25, a prompt on a frontier model) and state what would make you move up: GBDT for tabular (3.9 Ensembles: bagging, random forests and gradient boosting), a fine-tuned encoder for classification (7.5 Encoder-only: BERT and friends (and sentence embeddings)), a two-tower model for retrieval (9.2 Embeddings deep dive), a fine-tuned small LLM when prompting plateaus (8.2 Fine-tuning and PEFT). Explain the offline metric and why it tracks the business metric.
- Serving. Batch vs online, where the model runs, feature or context computation, caching, fallbacks and timeouts, rate limits, model versioning and rollback. For LLMs: prompt caching, streaming, routing between models, structured outputs (9.6 Generation side: context assembly, citations, structured outputs and tool calling).
- Evaluation. Offline evaluation on a held-out set with a baseline; online evaluation with A/B tests and guardrail metrics; for LLM systems, a golden set plus a calibrated LLM judge (9.8 Evaluating and observing LLM apps, 8.7 Evaluating LLMs) and human review of a sample.
- Monitoring and iteration. Data and prediction drift, latency and error dashboards, feedback loops (thumbs up/down, escalations), retraining or re-prompting cadence, incident playbooks, and the loop back to step 1 when requirements change.
Worked case 1: search or recommendation ranking
Prompt: design the "recommended for you" feed for a media app with 50 million users and 10 million items.
Requirements. Maximize long-term engagement (not just clicks), p99 latency under 200 ms, hundreds of thousands of requests per second at peak, items change daily, new users every day.
Data. Implicit feedback (watch time, skips) is abundant but biased by what was shown; log exposures and positions to correct for presentation bias. Split by time, not randomly, to avoid leakage from the future.
Model. A two-stage design. Candidate generation narrows 10 million items to a few hundred with cheap retrieval: a two-tower model that embeds users and items into the same space, trained with in-batch negatives (the InfoNCE idea from 5.8 Self-supervised and contrastive learning and 9.2 Embeddings deep dive), served with approximate nearest-neighbour search such as HNSW (9.3 Vector search and indexes); plus simple sources like "popular now" and "from creators you follow". Ranking scores the few hundred candidates with a richer model that sees cross features: a GBDT or a neural ranker trained on multiple objectives (watch, like, share) with a weighted combination tuned to the business goal. Address cold start with content features and exploration (a small random slice).
Serving. Precompute item embeddings and refresh daily; compute the user embedding at request time from recent activity; cache candidate lists for a few minutes; fall back to popularity if the ranker times out.
Evaluation. Offline: recall@k for candidate generation, nDCG and AUC for ranking against time-split logs. Online: A/B test on retention and watch time with guardrails on reports and diversity. Beware offline–online gaps caused by feedback loops.
Monitoring. Embedding drift, feature freshness, ranker calibration, diversity and creator fairness metrics; weekly retraining with automated evaluation gates.
Worked case 2: an LLM assistant with RAG for customer support
Prompt: design an assistant that answers customer questions from 20,000 help articles and tickets, escalating to humans when needed.
Requirements. Faithful answers with citations, p95 under 4 s, cost under a few cents per conversation, no leakage of other customers' data, measurable deflection rate without hurting satisfaction.
Data. Articles and resolved tickets, parsed and chunked with structure awareness (9.4 Chunking and ingestion); a golden set of 200 real questions with reference answers built with support staff; access-control metadata on every chunk.
Model. Start with a frontier model behind a grounded prompt. Retrieval: hybrid BM25 plus dense embeddings fused with RRF, then a cross-encoder reranker to the top five (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics). Generation: a prompt that demands citations and an explicit "I do not know; escalating" path; structured output for actions (refund request, ticket creation) through tool calls (9.6 Generation side: context assembly, citations, structured outputs and tool calling, 10.2 Tool use and function calling). Fine-tune a smaller model only when the golden-set results plateau and cost demands it (8.2 Fine-tuning and PEFT).
Serving. Prompt caching for the static instructions, streaming to keep perceived latency low, a cheap router that sends simple FAQ questions to a small model, timeouts with a fallback to search results, and per-tenant retrieval filters enforced in the vector store, not in the prompt.
Evaluation. Retrieval recall@5 on the golden set; faithfulness and answer relevance with a judge calibrated against staff labels (9.8 Evaluating and observing LLM apps); an adversarial set for prompt injection through retrieved documents (10.10 Agent safety and security); online: deflection rate, satisfaction, escalation precision.
Monitoring. Retrieval hit rate, refusal rate, judge scores on a daily sample, cost per conversation, new-question clustering to find gaps in the knowledge base, and a human-review queue for low-confidence answers.
The five interview question families
| Family | What is asked | How to prepare |
|---|---|---|
| Fundamentals | Bias–variance, regularization, metrics (precision vs recall, AUC), gradient descent variants, backprop by hand, attention equations, why LayerNorm, overfitting diagnosis | Re-derive from the lessons: 3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 3.6 Regularization and generalization, 4.4 Backpropagation, fully derived, 4.5 Optimizers and learning-rate schedules, 7.2 Scaled dot-product and multi-head attention, fully derived; practice explaining in two minutes with one equation |
| Coding | Implement kNN, k-means, logistic regression, attention or a tokenizer in NumPy; data-structure problems at medium difficulty; debugging a training loop | Projects 2, 3 and 7 in 11.3 Portfolio project ladder; time-boxed practice; narrate your approach |
| ML system design | The two cases above and variants: fraud detection, ad ranking, ETA prediction, content moderation | Rehearse the six-step framework aloud; keep a numbers sheet (latencies, QPS, costs, dataset sizes) |
| LLM engineering | RAG vs fine-tuning vs long context, chunking, embeddings, evals, hallucination mitigation, agent reliability, prompt injection defenses, cost control | 9.1 Why RAG, and the canonical pipeline–9.9 Production: latency, cost, caching, guardrails, security and privacy, 10.9 Evaluating agents–10.11 Building production agents; be ready with numbers from your own projects |
| Behavioral and projects | Walk me through a project; a time you were wrong; how you make trade-offs | One story per project in STAR form (situation, task, action, result) with a number and a failure |
A six-week preparation plan
Weeks 1–2: fundamentals, one hour a day re-deriving from the flashcards and lessons; weeks 3–4: coding drills and the two worked cases rehearsed aloud with a friend; week 5: LLM engineering questions answered from your projects; week 6: mock interviews, two per week, and a one-page cheat sheet of numbers. Throughout, keep the project READMEs from 11.3 Portfolio project ladder polished; they are the material interviewers will ask about.
Communication habits that score
Ask two or three clarifying questions first. State assumptions and numbers explicitly. Propose a baseline, then improve it, saying what evidence would trigger each upgrade. Name the trade-off at each decision (latency vs accuracy, cost vs quality, freshness vs stability). Finish with evaluation and monitoring; candidates who stop at "train the model" leave the strongest points on the table. When you do not know, say what you would measure to find out.
Where this shows up later
- The evaluation thinking here comes straight from 9.8 Evaluating and observing LLM apps and 10.9 Evaluating agents; the serving concerns from 9.9 Production: latency, cost, caching, guardrails, security and privacy and 10.11 Building production agents.
- Staying current (11.5 Staying current) keeps your numbers sheet accurate: model prices, context lengths and benchmark leaders change every few months.
- The frontier topics in 11.6 Frontier 2026: where the field is heading are increasingly asked in LLM-engineering rounds.
Common pitfalls (4)
- Jumping to a model before clarifying requirements; designing the wrong system well is still a fail.
- Random train/test splits on time-ordered data, which leaks the future and inflates offline metrics.
- Enforcing access control in the prompt instead of in the retrieval layer.
- Describing a design without numbers; interviewers cannot evaluate 'fast' or 'cheap'.
Recap: all 7 key points
- Six steps, said aloud: requirements → data → model → serving → evaluation → monitoring.
- Turn goals into numbers (latency percentiles, cost per request, target metric) before proposing a model.
- Recommendation and search use two stages: cheap candidate generation (two-tower embeddings + ANN) and a richer ranker; evaluate with recall@k, nDCG and time-split A/B tests.
- An LLM assistant with RAG needs hybrid retrieval + reranking, a cited grounded prompt with a refusal path, tool calls for actions, and access control enforced in the store.
- Evaluate LLM systems with a golden set, a judge calibrated against humans and an adversarial injection set; monitor refusal rate, judge scores and cost.
- Five question families: fundamentals, coding, ML system design, LLM engineering, behavioral; prepare each from your own projects.
- Always start from a baseline and say what evidence would justify each upgrade; finish with evaluation and monitoring.
Watch
The training pipeline and practical guidance that system-design interviews assume you know.
Read
Exercises
- Two designs, aloud
Record yourself designing (a) a fraud-detection system for card payments and (b) a documentation assistant for an API product, using the six steps, in 25 minutes each. Expected: you state at least five numbers, one baseline and one upgrade trigger per design, and you finish with monitoring.
- Numbers sheet
Build a one-page sheet: typical latencies (vector search, reranker, LLM time-to-first-token), token prices for two frontier and two small models, context lengths, embedding dimensions, HNSW memory per vector, QPS a single GPU can serve. Expected: every number sourced and dated; refresh it monthly (11.5 Staying current).
- Fundamentals sprint
Answer in writing, two minutes each: derive the bias–variance decomposition; write attention and explain ; explain why AUC can be high while precision is low; show the gradient of logistic regression. Expected: four correct derivations checked against 3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 7.2 Scaled dot-product and multi-head attention, fully derived, 3.3 Logistic regression and softmax classification.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →