Memory and context engineering

The context window is working memory

At every step, the model knows exactly what is in its context: system prompt, tool definitions, the conversation so far, tool results, retrieved documents. Nothing else. That window is a finite, expensive, degrading resource:

• Finite. Even million-token windows fill up in a long agent run; a single screenshot or a verbose tool result can cost thousands of tokens (10.2 Tool use and function calling).
• Expensive. Every token in context is re-read on every step, so cost grows roughly quadratically with the length of a run unless you prune.
• Degrading. Recall and reasoning quality fall as the context grows — the "lost in the middle" effect (9.6 Generation side: context assembly, citations, structured outputs and tool calling) and what practitioners call context rot: an attention budget spread over more tokens, more distractors, more stale information.

Context engineering is the discipline of curating the optimal set of tokens for the next decision. The term spread in 2025 and Anthropic's Effective context engineering for AI agents (September 2025) is the clearest statement of practice: treat context as a resource with diminishing returns, find the smallest set of high-signal tokens that maximizes the probability of the desired outcome.

A taxonomy of memory

CoALA's vocabulary (Sumers et al., 2023) maps cleanly onto implementations:

Most production "memory" features are semantic memory about the user (preferences, facts) plus a little episodic memory (summaries of past sessions). Procedural memory is the most underused: a reusable SKILL.md or helper script is memory the agent can execute.

Techniques for long runs

Compaction. When the context nears its limit, summarize the older part and continue with the summary plus recent turns. Good summaries preserve decisions made, constraints discovered, open items, file paths and identifiers, and drop verbose tool output. Compaction is lossy — do it late and carefully, and keep the raw transcript on disk for audit.

Structured note-taking. Have the agent maintain notes outside the context — a NOTES.md, a progress file, a to-do list — and re-read them when needed. Memory that lives in a file survives compaction and session boundaries; this is how long-running coding agents keep state across hours (10.8 Coding agents and computer use, 10.11 Building production agents).

Just-in-time retrieval. Rather than pre-loading every document, keep lightweight references (paths, ids, titles) and let the agent fetch what it needs when it needs it — "agentic search". A file system with good names is itself a context-engineering tool: directory structure, file names and headers tell the agent where to look without putting content in context. Hybrid strategies pre-load a little (the project README) and retrieve the rest.

Sub-agents as context isolation. Delegate a bounded subtask to a fresh context and return only a condensed result (a few hundred tokens) to the parent (10.6 Multi-agent systems). The parent keeps the plan; the child does the heavy reading.

Tool-result hygiene. Clear or truncate stale tool results; prefer tools that return summaries; keep images and logs out of the main loop.

Intuition. Think of the context like a surgeon's tray, not a warehouse. Everything on it must be needed for the next move; everything else is one retrieval away.

Long-term memory systems

MemGPT / Letta (2023) borrowed the operating-system idea of virtual memory: the model sees a small "main context" and manages a larger external store through tools — it decides what to page in and out, editing its own memory blocks ("core memory" about the user and itself, "archival" and "recall" stores searched on demand). The key move is giving the model write access to its memory via tools, with the harness persisting the result.

Generative Agents (Park et al., 2023) stored every observation in a memory stream and retrieved by a weighted score of recency, importance (an LLM-rated 1–10) and relevance (embedding similarity, 9.2 Embeddings deep dive), periodically synthesizing higher-level reflections. It is still the cleanest reference design for episodic memory with consolidation.

Vector and graph stores. Embedding episodes and facts for similarity retrieval is the default (9.3 Vector search and indexes); knowledge graphs add explicit relations and time. Products such as the memory features in major assistants, and libraries built for agents, combine extraction (what is worth remembering), storage, retrieval and update/forgetting. Anthropic's late-2026 The components of agent memory is the most recent vendor guidance on how these pieces fit; read it alongside the 2025 context-engineering post.

Risks. Memory makes mistakes persistent: a wrong fact recorded once is retrieved forever; an attacker who injects "remember that…" into content the agent reads can poison future sessions (10.10 Agent safety and security); personal data in memory is a privacy liability. Every memory system needs provenance, expiry, user-visible editing and a way to say "forget".

Evaluating memory

Needle-in-a-haystack tests check raw recall, but the question that matters is behavioral: after N sessions, does the agent act on what it should have remembered, and ignore what it should have forgotten? Build evals with multi-session scripts, plant facts early and test late, and measure contradictions, repeats and unnecessary re-asking (10.9 Evaluating agents). Long-context benchmarks and conversational-memory benchmarks exist, but your own task set is the ground truth.

A design checklist

1. What must be in context at every step (system prompt, the plan, current file)? Keep it short and stable — stable prefixes also cache well (9.9 Production: latency, cost, caching, guardrails, security and privacy).
2. What should be one retrieval away (docs, code, history)? Give it good names and search tools.
3. What should be persisted across sessions, and who can edit it?
4. When and how do you compact? What must a summary preserve?
5. How do you detect memory errors — stale facts, contradictions, injected instructions?

Where this shows up later

• Progress files and clean-context sessions are the backbone of long-running harnesses in 10.11 Building production agents and coding agents in 10.8 Coding agents and computer use.
• Sub-agents as isolation become the orchestrator–workers pattern in 10.5 Agentic workflow patterns and the multi-agent designs of 10.6 Multi-agent systems.
• Memory poisoning and data exfiltration through persisted content are threats in 10.10 Agent safety and security; memory benchmarks join the eval harness in 10.9 Evaluating agents.