Frameworks and SDKs
What LangChain, LlamaIndex, DSPy and plain SDKs buy you, and what they cost
Every RAG tutorial is written in some framework, and every framework hides decisions you need to understand. Knowing what LangChain and LangGraph, LlamaIndex, Haystack, DSPy and the thin provider SDKs actually provide — integrations, graph runtimes, typed outputs, prompt optimization — and what they cost in abstraction, churn and debuggability lets you choose deliberately, keep your evaluation harness independent, and drop a framework when it gets in the way. The same judgment carries into agent frameworks in Phase 10 (10.6 Multi-agent systems, 10.11 Building production agents).
You'll be able to
- Describe what each major framework is for and where it fits in a RAG system
- Weigh the benefits of frameworks (integrations, graph runtimes, tracing) against their costs (abstraction leakage, hidden prompts, churn)
- Assemble a minimal, framework-light stack for learning and a deliberate stack for production
- Explain DSPy's signature, module and optimizer model and when prompt optimization pays off
- Use framework source code as a library of worked solutions to learn from
Key ideas
- Frameworks provide integrations, pattern implementations, graph runtimes (state, checkpoints, human-in-the-loop) and tracing hooks.
- They cost abstraction leakage, hidden prompts, API churn, overhead and a distorted mental model of what the system actually does.
- Learn with the thin stack: provider SDK + embeddings + NumPy/FAISS + Pydantic + one tracing tool; graduate deliberately.
Show 4 more key ideas
- In production pick one orchestration layer for a reason, version prompts and schemas, and keep the eval harness framework-independent.
- DSPy's signatures, modules and optimizers compile prompts against a metric — prompt engineering becomes optimization once a golden set exists.
- 12-factor agents: own your prompts, context window and control flow; tools are structured outputs; small composable components.
- Read framework source as worked solutions, then reimplement the pattern yourself in thirty lines.
Suggested path through this lesson
The landscape
| Tool | What it is | Strengths | Costs and risks | Best for |
|---|---|---|---|---|
| LangChain + LangGraph | Integrations library plus a graph runtime for stateful, multi-step apps | huge ecosystem; persistence, retries and human-in-the-loop pauses; LangSmith tracing | abstraction churn; hidden prompts in legacy chains | complex workflows, agents, teams already using it |
| LlamaIndex | Data framework for LLM apps | best-in-class loaders, node parsers, indices, query engines, agentic RAG | many concepts to learn; some magic defaults | document-heavy RAG and ingestion |
| Haystack | Pipeline framework from deepset | explicit, typed pipelines; production focus | smaller ecosystem | production search and RAG pipelines |
| DSPy | Declarative modules plus optimizers that tune prompts against a metric | turns prompt hacking into optimization; pairs with evals | needs a metric and data; a different mental model | pipelines you can measure |
| Provider SDKs (Anthropic, OpenAI, Google) | Thin clients: messages, tools, structured outputs, files, caching | full control, no hidden prompts, newest features first | you write the glue | learning; most production systems |
| Instructor and Pydantic AI | Typed outputs and typed agents on top of SDKs | validation, retries, minimal magic | opinionated | structured extraction, typed tools |
| Vercel AI SDK | TypeScript toolkit with streaming UI hooks | excellent front-end integration | TypeScript-centric | web applications |
Agent-specific SDKs — the OpenAI Agents SDK, the Claude Agent SDK, Google's ADK, smolagents, CrewAI, AutoGen — belong to Phase 10 (10.6 Multi-agent systems, 10.11 Building production agents).
What a framework buys you
Hundreds of integrations (document loaders, vector stores, model providers), tested implementations of patterns (parent–child retrieval, reciprocal rank fusion, adaptive RAG), a graph runtime with state, checkpoints, retries and human-in-the-loop pauses, tracing hooks, and a community whose examples you can read. When you need to swap a vector store or try a reranker, a framework turns a day into an hour.
What it costs you
Abstraction leakage: when something goes wrong you debug the framework and the model. Hidden prompts: default templates you did not write shape behaviour and change between versions. Churn: APIs evolve quickly and tutorials rot in months. Performance: layers of wrappers add latency and make token accounting opaque. Lock-in of your mental model: you stop seeing that a RAG pipeline is fifty lines of Python (9.6 Generation side: context assembly, citations, structured outputs and tool calling). The 12-factor agents principles distilled industry experience into advice that holds for RAG too: own your prompts, own your context window, own your control flow; tools are just structured outputs; keep components small and composable.
Intuition. A framework is a rental car: perfect for a week in a new city, expensive and limiting if you live there. Learn to drive first.
A recommended stack
For learning (this phase and the next): the provider SDK, sentence-transformers or an embeddings API, NumPy or FAISS, Pydantic for typed outputs, and one tracing tool — nothing else until you can explain every call. For production: one orchestration layer chosen deliberately (a graph runtime when you have branching, retries and human approvals; LlamaIndex when ingestion is the hard part; the plain SDK when the flow is simple), prompts and schemas in version control, an eval harness (9.8 Evaluating and observing LLM apps) that is independent of the framework so you can switch, and tracing from day one. Add DSPy when you have a metric and want the machine to tune prompts and few-shot examples.
DSPy: programming, not prompting
DSPy replaces hand-written prompts with signatures (typed input→output declarations such as question, context -> answer), modules that implement strategies (predict, chain-of-thought, ReAct), and optimizers that search over instructions and few-shot demonstrations to maximize a metric on your training set. The result is a compiled program whose prompts are artifacts you can inspect and version. It formalizes the loop this phase keeps repeating — define a metric, build a dataset, optimize against it — and it is a natural fit once your golden set (9.8 Evaluating and observing LLM apps) exists.
Reading framework code to learn
The best use of frameworks for a learner is as a library of worked solutions. Read how LlamaIndex implements sentence-window and parent–child retrieval, how LangChain's retrievers compose, how a graph runtime persists state between steps — then write your own thirty-line version. You will understand the trade-offs, and you will be able to drop the framework when it gets in the way.
A decision procedure
- Can you write it in under 200 lines with the SDK? Then do that first; it is your baseline and your fallback.
- Do you need branching, retries, checkpoints or human approval mid-flow? Consider a graph runtime.
- Is ingestion (parsers, chunkers, metadata extraction) the hard part? Consider LlamaIndex.
- Do you have a metric and a dataset and want prompts tuned automatically? Consider DSPy.
- Whatever you choose, keep prompts, schemas and the eval harness outside the framework.
Where this shows up later
- Agent frameworks and SDKs — 10.6 Multi-agent systems, 10.11 Building production agents; MCP as the integration layer that makes tool plumbing framework-agnostic — 10.7 MCP and interoperability (plus Agent Skills and A2A).
- The eval harness that must outlive your framework choice — 9.8 Evaluating and observing LLM apps; the capstone uses the learning stack — 9.11 Project: a cited RAG assistant over your own documents, with evals.
Common pitfalls (4)
- Learning RAG through a framework's abstractions and never seeing the forty-line version underneath.
- Coupling your evaluation harness to a framework so that switching costs you your measurements.
- Shipping default prompt templates you have never read.
- Adopting a graph runtime for a linear retrieve-then-answer flow that needs none of its features.
Recap: all 7 key points
- Frameworks provide integrations, pattern implementations, graph runtimes (state, checkpoints, human-in-the-loop) and tracing hooks.
- They cost abstraction leakage, hidden prompts, API churn, overhead and a distorted mental model of what the system actually does.
- Learn with the thin stack: provider SDK + embeddings + NumPy/FAISS + Pydantic + one tracing tool; graduate deliberately.
- In production pick one orchestration layer for a reason, version prompts and schemas, and keep the eval harness framework-independent.
- DSPy's signatures, modules and optimizers compile prompts against a metric — prompt engineering becomes optimization once a golden set exists.
- 12-factor agents: own your prompts, context window and control flow; tools are structured outputs; small composable components.
- Read framework source as worked solutions, then reimplement the pattern yourself in thirty lines.
Watch
LLMs as a new kind of computer and 'partial autonomy' apps; context for why thin, inspectable tooling matters.
Read
Exercises
- Same pipeline, two ways
Re-implement the minimal pipeline from 9.6 Generation side: context assembly, citations, structured outputs and tool calling with LlamaIndex (or LangChain). Compare lines of code, latency per query, how easy it is to see the final prompt, and what the framework did that you did not ask for. Write five sentences on which you would choose for a production system and why.
- Read the source
Open the implementation of sentence-window (or parent–child) retrieval in LlamaIndex and write your own version in under forty lines using your existing chunker and embeddings. Verify both return the same parents for ten queries.
- Compile a prompt with DSPy
Define a DSPy signature for grounded QA, a module with retrieval plus chain-of-thought, and a metric from your golden set (9.8 Evaluating and observing LLM apps). Run an optimizer and compare the compiled program's score and prompt with your hand-written version.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →