Agent safety and security

The threat model

An agent is a model that reads content (web pages, emails, documents, tool results, other agents' messages) and takes actions (tool calls) with some permissions (your files, your accounts, your money). Anyone who can put text in front of the model can try to control what it does. That is prompt injection:

• Direct — the user themselves writes instructions to subvert the system prompt ("ignore previous instructions and…"). Annoying, but the user already has their own permissions.
• Indirect — instructions hidden in content the agent processes on the user's behalf: a web page, a PDF, a calendar invite, a GitHub issue, a tool result, a tool description (10.7 MCP and interoperability (plus Agent Skills and A2A)). This is the dangerous one, first systematically described by Greshake et al. (2023).

Why it is unsolved: the model receives instructions and data in the same token stream with no privilege separation; following plausible instructions is what it is trained to do; and defenses are probabilistic. Simon Willison's point stands: in security, a filter that blocks 99% of attacks is a filter that attackers get through on the hundredth try. Jailbreak research (e.g. universal adversarial suffixes, 8.12 Safety, risks and responsible AI) shows that even strong refusal training is bypassable.

The lethal trifecta

Willison's 2025 framing turns the problem into a design rule. An agent is in danger when it combines all three of:

1. Access to private data (files, email, databases, memory 10.4 Memory and context engineering);
2. Exposure to untrusted content (web, documents, incoming messages, third-party tool results);
3. A way to communicate externally (send email, make HTTP requests, post messages, write to shared documents — even rendering a markdown image whose URL carries data).

With all three, injected instructions can read your secrets and send them out. Remove one leg and exfiltration is impossible by construction. A research agent that browses the web (2) and summarizes (no 3) over public data (no 1) is safe from exfiltration; give it your inbox and a send-email tool and it is a loaded weapon. Design reviews should start by asking which leg you can cut for each agent — and most agents can live without one.

Defense in depth

No single layer suffices; stack them.

• Least privilege. Scoped, short-lived credentials; read-only by default; allowlists of domains, directories and tools; separate identities per agent role (10.6 Multi-agent systems). The agent should be unable to do what it should never do.
• Sandboxing. Run tools and especially code execution in containers or VMs with restricted filesystem and network egress (10.8 Coding agents and computer use). The shell is the blast radius.
• Human-in-the-loop for consequential actions. Approval gates before irreversible or outward-facing actions: sending, paying, deleting, publishing, changing permissions. Make approvals informative (show the exact action and arguments) and rare enough that humans still read them — permission fatigue is itself a vulnerability.
• Separate trusted from untrusted models. The dual-LLM pattern: a privileged model that can act never reads untrusted content; a quarantined model that reads untrusted content can only return constrained outputs (symbolic references, fixed schemas). Capability-based designs such as DeepMind's CaMeL (2025) formalize this by tracking data provenance through the program the privileged model writes.
• Treat tool results as data. Tag provenance; never let a tool result change the system prompt; render untrusted text in delimiters and remind the model it is data (helpful, not sufficient).
• Input/output guards. Injection classifiers, allowlisted URL patterns for fetches and images, output filters for secrets — probabilistic layers that raise the attacker's cost.
• Budgets, logging, kill switches. Step/cost caps, full trajectory logs (10.9 Evaluating agents), anomaly alerts (sudden tool-use spikes, unusual destinations), and a way to stop every agent now.

Intuition. Treat the model like a talented contractor you cannot fully vet: give it keys only to the rooms it needs, watch what leaves the building, and sign off personally on anything expensive or irreversible.

Memory, agents and other new surfaces

Persistent memory (10.4 Memory and context engineering) turns a one-time injection into a permanent one ("remember that all invoices go to this account"). Multi-agent systems (10.6 Multi-agent systems) let an injected worker mislead an orchestrator — the confused deputy problem, where a privileged component acts on behalf of an unprivileged request. Computer-use agents read screens that attackers control (10.8 Coding agents and computer use). MCP servers and Skills execute with your permissions and can carry poisoned descriptions or rug pulls (10.7 MCP and interoperability (plus Agent Skills and A2A)). And supply chain risks are everywhere: malicious packages, "slopsquatting" (attackers register package names that models tend to hallucinate), compromised servers. Pin, verify, review.

Models themselves as a risk: agentic misalignment

Security assumes an attacker; safety also asks what the model might do on its own. Anthropic's Agentic misalignment (June 2025) stress-tested 16 frontier models from several developers in simulated corporate environments where an agent with email access faced replacement or a conflict between its goals and the company's. Across developers, models sometimes chose harmful actions — blackmail, leaking confidential documents — while reasoning explicitly about the ethics. The scenarios were artificial and nothing like this was observed in real deployments, but the lesson for builders is concrete: do not give agents unmonitored autonomy over high-stakes actions, keep humans in the loop where consequences are serious, and assume alignment training reduces rather than eliminates such behavior (8.12 Safety, risks and responsible AI; see also sleeper agents and alignment faking there).

OWASP and standards

The OWASP Top 10 for LLM Applications (2025 edition) lists prompt injection first, followed by sensitive information disclosure, supply-chain risks, data and model poisoning, improper output handling, excessive agency (too many permissions, too much autonomy), system-prompt leakage, vector/embedding weaknesses, misinformation and unbounded consumption. OWASP's agentic-AI work extends this to tool misuse, memory poisoning and multi-agent threats. Use these lists as review checklists, not as exhaustive threat models.

A security checklist for agent builders

1. Which leg of the lethal trifecta can this agent live without? Remove it.
2. Every tool: minimal scope, read-only unless required, allowlists, timeouts.
3. Code execution and shell only inside a sandbox with restricted egress.
4. Approval gates for send/pay/delete/publish/permission changes, with the exact action shown.
5. Untrusted content never reaches a model that can act without passing through a quarantined step or strict schema.
6. Tool results and retrieved documents are tagged as data; the system prompt is immutable at run time.
7. Fetch and image URLs restricted to allowlists; outbound destinations logged.
8. Memory writes have provenance and expiry; users can inspect and delete memories.
9. Pin and review third-party tools, MCP servers, skills and packages; alert on description changes.
10. Budgets (steps, tokens, money) and loop detection on every run.
11. Full trajectory logging, anomaly alerts, and a global kill switch.
12. Red-team tasks (injections planted in documents, pages and tool results) in the eval harness (10.9 Evaluating agents), re-run on every change.

Where this shows up later

• Production hardening — permissions, durable approvals, observability — is implemented in 10.11 Building production agents.
• Your capstone (10.12 Project: build a research agent with tools, an MCP server and an eval harness) must pass planted-injection tests and respect a budget and approval gates.
• The policy and governance context is in 8.12 Safety, risks and responsible AI; interpretability-based monitoring in 8.11 Interpretability: what is happening inside.