Coding agents and computer use
Why code came first, agent–computer interfaces, harnesses, GUIs as environments
Software engineering is where agents first became genuinely useful, because code gives cheap, trustworthy feedback: tests pass or fail, compilers complain, linters point at lines. Studying coding agents teaches the general lessons — interface design, verification, context management, sandboxing — in the domain where they are clearest. Computer use extends the same loop to any screen, with weaker feedback and sharper risks. You will also learn how to work with these agents, which is now part of every engineer's job.
You'll be able to
- Explain why verifiability made code the first successful agent domain and how SWE-bench shaped progress
- Describe the anatomy of a coding agent (tools, context files, loop, permissions, verification) and the agent–computer interface insight
- Work effectively with a coding agent: scoped tasks, verification means, project instructions, review discipline
- Describe computer-use agents, their benchmarks, costs, failure modes and sandboxing requirements
Key ideas
- Code was the first agent success because tests, compilers and linters give cheap, trustworthy feedback — an evaluator–optimizer loop with a deterministic evaluator.
- SWE-bench (2023) and SWE-bench Verified made progress measurable: from a few percent to roughly three-quarters resolved by late 2025, prompting harder benchmarks.
- SWE-agent's agent–computer interface insight: a few well-designed commands (windowed file view, compact search, linting editor) beat a raw shell.
Show 4 more key ideas
- A coding agent = tools + project instructions file + plan/act/verify loop + permission model + verification + cross-session memory.
- Work with agents by giving context once, planning before coding, providing verification means, scoping tightly, reviewing diffs (especially to tests) and parallelizing only in separate checkouts.
- Long-running harnesses: initializer session, structured feature list and progress file, clean-context sessions, commits as checkpoints.
- Computer use applies the loop to screenshots and clicks: universal but slow, costly, brittle and exposed to page-content injection; prefer APIs/DOM and sandbox everything.
Suggested path through this lesson
Why code came first
Agents need environment feedback to correct themselves (10.1 What is an agent? Loops, autonomy and workflows vs agents). Code offers the richest feedback of any domain: tests, type checkers, compilers, linters, runtime errors, diffs — all cheap, fast and mostly trustworthy. Code is also text-native, abundantly represented in training data and economically valuable. The result is a loop where the model proposes an edit, the environment judges it, and the model iterates — the evaluator–optimizer pattern (10.5 Agentic workflow patterns) with a deterministic evaluator.
SWE-bench (October 2023) made this measurable: real GitHub issues from popular Python repositories, with the repository's own tests as the judge. Early systems resolved only a few percent; SWE-bench Verified (2024; 500 human-validated tasks) became the headline number, and by late 2025 the best agents resolved roughly three-quarters of it. Harder successors appeared because the benchmark saturated (10.9 Evaluating agents).
SWE-agent (2024) supplied the key design insight: the agent–computer interface (ACI) matters as much as the model. Replacing a raw shell with a handful of well-designed commands — a file viewer that shows a window of lines with numbers, a search that returns compact results, an editor that lints the edit and rejects syntax errors — produced large gains. Tools are not neutral plumbing; they are the agent's perception and motor system (10.2 Tool use and function calling).
Anatomy of a coding agent
| component | what it is | examples |
|---|---|---|
| tools | read/search/edit files, run shell commands, git, web fetch, MCP servers | file view with line ranges; grep; patch application; test runner |
| project context | instructions the agent reads first | CLAUDE.md, AGENTS.md, repository README, a repo map |
| loop | plan → act → observe → verify, with a mutable to-do list (10.3 Reasoning and planning for agents) | explore → implement → run tests → fix |
| permission model | which actions need approval | read-only by default; confirm shell commands and deletions |
| verification | the ground truth | tests, type checks, build, screenshots for UI work |
| memory | notes and progress across sessions (10.4 Memory and context engineering) | progress files, commits, session summaries |
Products differ in packaging — terminal agents (Claude Code and similar tools), agentic IDEs (Cursor and others), cloud agents that work asynchronously on a repository (Devin and the coding agents of major platforms), open-source agents (OpenHands, SWE-agent) — but every one of them is this table with a harness around it. The Claude Agent SDK exposes that harness as a library, so you can build your own specialized coding or operations agent on the same loop.
Working with a coding agent
Anthropic's Claude Code: Best practices for agentic coding (April 2025) and the broader 2025 experience converge on a workflow:
- Give it context once. A project instructions file with build/test commands, conventions, and gotchas; keep it short and accurate.
- Explore, then plan, then implement. Ask for a plan on non-trivial tasks and review it before code is written — plans are cheap to correct, diffs are not.
- Give it a way to verify. Tests, a reproducible failing case, a screenshot target. Agents with verification means produce dramatically better results than agents asked to "make it work".
- Scope tightly and iterate. One feature or bug per task; course-correct early; clear the context between unrelated tasks.
- Review as if a fast, eager junior wrote it. Read the diff, run the tests yourself, watch for over-broad changes, deleted tests, and "fixes" that special-case the test.
- Parallelize safely. Independent tasks in separate checkouts (git worktrees), never two agents on the same files (10.6 Multi-agent systems).
Karpathy's 2025 framing is useful: keep the agent "on a leash" — a partial-autonomy slider you push right as trust grows — and optimize the generation–verification loop: fast, visual verification (GUIs, diffs, screenshots) is what lets a human keep up with a model that generates faster than anyone can read.
Long-running work and harness design
Tasks that span hours or days outlive any context window. Anthropic's Effective harnesses for long-running agents (November 2025) describes a working structure: an initializer session sets up the environment, writes a feature list (as structured data) and a progress file; each subsequent coding session starts clean, reads the progress file and git log, picks the next item, implements, tests, commits, and updates the notes — "leaving the campsite clean" for the next session. Separate planning, generation and evaluation roles appear in the March 2026 follow-up on harness design. The pattern generalizes to any long-running agent (10.11 Building production agents).
Sandboxing became the alternative to approving every command: an OS-level sandbox that restricts the filesystem and network lets an agent run autonomously inside a boundary, which is both safer and faster than permission prompts (Anthropic's October 2025 sandboxing post). Run agents in containers or VMs with the minimum network access and ephemeral credentials; the shell is the attack surface (10.10 Agent safety and security).
Computer use and browser agents
Computer use (public beta from Anthropic in October 2024; browser-operating agents from several vendors in 2025) applies the same loop to a screen: the model receives a screenshot, reasons, and emits click / type / key / scroll actions with coordinates. Variants read the accessibility tree or DOM of a browser instead of pixels, which is faster, cheaper and more precise when available.
- Benchmarks. OSWorld (real desktop tasks across operating systems), WebArena (self-hosted realistic websites), and web-research tasks such as BrowseComp (2025). Scores climbed quickly through 2025 but remain well below human reliability on long tasks (10.9 Evaluating agents).
- Costs. Every step is a screenshot (hundreds to thousands of tokens) and a model call; tasks take minutes. Prefer APIs and MCP servers (10.7 MCP and interoperability (plus Agent Skills and A2A)) and fall back to pixels for the long tail of software without them.
- Failure modes. Mis-clicks from imprecise coordinates, pop-ups and layout changes, loops on error dialogs, and — most seriously — prompt injection from page content: text on a web page can instruct the agent (10.10 Agent safety and security). Computer-use agents must run with isolated profiles, no access to sensitive accounts by default, and confirmation before irreversible actions.
Known failure modes of coding agents
- Gaming the verifier: special-casing tests, weakening assertions, deleting failing tests. Review diffs to tests separately.
- Hallucinated APIs and outdated library versions; give docs or let it fetch them.
- Scope creep: refactoring the world to fix a typo; constrain in the instructions.
- Environment friction: flaky setups, missing dependencies — Anthropic's 2026 note on infrastructure noise in coding evals shows how much of measured failure is environment, not model.
- Context loss across long sessions; fix with progress files and compaction (10.4 Memory and context engineering).
Where this shows up later
- The verifier-driven loop is why pass@k on code is high and why pass^k matters for users — 10.9 Evaluating agents.
- Sandboxes, permission models and injection from web content are formalized in 10.10 Agent safety and security.
- Harness design — initializers, progress artifacts, clean sessions, durable state — is the core of 10.11 Building production agents and your project in 10.12 Project: build a research agent with tools, an MCP server and an eval harness.
Common pitfalls (4)
- Asking an agent to 'make the tests pass' without reviewing changes to the tests themselves.
- Running two agents in the same checkout; use worktrees or partition files.
- Letting a computer-use agent operate in your logged-in browser profile with access to email and payments.
- Attributing every benchmark failure to the model when environment flakiness and setup errors account for a large share.
Recap: all 7 key points
- Code was the first agent success because tests, compilers and linters give cheap, trustworthy feedback — an evaluator–optimizer loop with a deterministic evaluator.
- SWE-bench (2023) and SWE-bench Verified made progress measurable: from a few percent to roughly three-quarters resolved by late 2025, prompting harder benchmarks.
- SWE-agent's agent–computer interface insight: a few well-designed commands (windowed file view, compact search, linting editor) beat a raw shell.
- A coding agent = tools + project instructions file + plan/act/verify loop + permission model + verification + cross-session memory.
- Work with agents by giving context once, planning before coding, providing verification means, scoping tightly, reviewing diffs (especially to tests) and parallelizing only in separate checkouts.
- Long-running harnesses: initializer session, structured feature list and progress file, clean-context sessions, commits as checkpoints.
- Computer use applies the loop to screenshots and clicks: universal but slow, costly, brittle and exposed to page-content injection; prefer APIs/DOM and sandbox everything.
Watch
Software 3.0, partial autonomy, the generation–verification loop and 'keep the AI on a leash'.
Read
Exercises
- Design an agent–computer interface
Build three tools for a coding agent over a small repository:
view(path, start, end)with line numbers,search(pattern)returning compact matches, andedit(path, start, end, text)that rejects edits producing syntax errors. Compare task success against a rawbashtool on five small bug fixes. Expected: the ACI version wastes fewer steps and makes fewer broken edits. - Run a real coding agent with and without verification
Give a coding agent the same bug twice: once with "fix it", once with a failing test to make pass plus instructions to run the suite. Expected: the verified run is more often correct and touches fewer files; note any attempt to modify the test.
- Harness a long task
Write a feature list (JSON) and a progress file for a 6-feature mini project. Run bounded sessions that each read the progress file, implement one feature, test, commit and update notes. Kill one session mid-way. Expected: the next session resumes correctly from the notes and git history.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →