Reasoning and planning for agents

Why planning matters

An agent's run is a path through a tree of possible actions. If the model picks the right branch with probability $p$ at each of $n$ decision points, the whole run succeeds with probability about $p^n$ (10.9 Evaluating agents). Planning techniques raise $p$ at the decisions that matter by spending more tokens there: thinking before acting, considering alternatives, checking results. The art is spending that compute where it changes the outcome and nowhere else.

ReAct: reason, then act, then look

ReAct (Yao et al., October 2022) interleaves free-form thoughts with actions and the observations they produce:

Question: Which city hosted the Olympics the year the first iPhone was released?
Thought 1: I need the iPhone release year, then the Olympics host that year.
Action 1:  search("first iPhone release year")
Observation 1: The first iPhone was released on June 29, 2007.
Thought 2: Now find the 2007 Olympics... there were no Olympics in 2007; Summer Games are 2004/2008.
           The question may be ill-posed; the nearest Games are Beijing 2008. Let me verify.
Action 2:  search("2008 Summer Olympics host city")
Observation 2: Beijing hosted the 2008 Summer Olympics.
Thought 3: Answer with the caveat.
Action 3:  finish("No Olympics were held in 2007; the next Games were Beijing 2008.")

Two mechanisms make this work. Reasoning traces let the model track a plan, handle exceptions and decide what to look up; actions ground the reasoning in external information, which cut hallucination compared with chain-of-thought alone (8.4 Prompting as programming). In a modern API the "Thought" is the model's text (or its extended-thinking channel) and the "Action" is a tool call (10.2 Tool use and function calling) — ReAct is simply the loop with reasoning made explicit.

Plan-and-execute

Instead of deciding one step at a time, the model first writes a plan (an ordered list or DAG of subtasks), then executes steps — often with cheaper calls — and replans when an observation contradicts the plan. Advantages: fewer high-end model calls, independent steps can run in parallel (10.5 Agentic workflow patterns), and the plan is a readable artifact for humans to approve. Disadvantage: plans go stale; a rigid executor marches on after step 2 invalidated step 5. Good implementations treat the plan as a mutable scratchpad — the to-do list a coding agent maintains and edits is exactly this (10.8 Coding agents and computer use).

Search over thoughts

When a single path is unreliable, search several:

Tree of Thoughts (2023) made a point that still holds: models can evaluate candidate partial solutions better than they can generate the right one in one pass, so generate-then-judge beats generate-only — if the judgment is reliable.

Reflection and self-critique

Reflexion (2023) runs a task, receives a signal (a failed test, a wrong answer), writes a short verbal reflection ("I assumed the list was sorted; check first"), stores it in an episodic memory, and retries with the reflections in context — "verbal reinforcement learning" without weight updates. Self-Refine loops generate → feedback → refine with the same model. The honest finding across these methods: they help a lot when the feedback comes from an external signal (unit tests, an interpreter, a search result, a human) and much less when the model critiques itself with nothing new to look at — models tend to agree with themselves and show self-preference bias (8.7 Evaluating LLMs).

Intuition. Verification is the bottleneck. A planner with a perfect verifier can brute-force its way to success (best-of-$N$ with $N$ large). A planner with no verifier is just hoping. Design tasks so that something in the environment can say "no".

Reasoning models as planners

Since late 2024, models trained with reinforcement learning on verifiable rewards (8.6 Reasoning models and test-time compute) "think" before answering and between tool calls. Denny Zhou's point from the Berkeley agents course is that reasoning is latent in pretrained models and can be elicited by decoding and prompting, but RL is what makes it reliable; Hyung Won Chung's framing is "don't teach — incentivize". For agents this changed the playbook:

• Hand-written CoT scaffolds matter less. Interleaved thinking between tool calls gives you ReAct for free; explicit "think step by step" prompts add little.
• Budgets became a dial. Thinking tokens cost money and latency; reasoning effort or thinking budgets let you spend more on hard steps and almost nothing on routine ones.
• Search moved inside. Much of what Tree of Thoughts did explicitly, a reasoning model does implicitly — but external verification and best-of-$N$ with a real verifier still add value.

When more deliberation hurts

Overthinking is real: on easy requests long reasoning adds latency, cost and sometimes errors (the model talks itself out of the right answer). Heuristics that work in practice:

• Easy, reversible, cheap-to-verify → act directly, low or no thinking budget.
• Hard decision, irreversible or expensive to redo → think more; consider alternatives; ask for confirmation (10.10 Agent safety and security).
• Verifier available → sample several candidates and verify rather than thinking longer once.
• No feedback at all → plan explicitly so a human can review the plan before execution.

Where this shows up later

• The plan as a mutable artifact becomes the progress file and to-do list of long-running agents in 10.4 Memory and context engineering and 10.11 Building production agents.
• Best-of-$N$ with verifiers and evaluator loops are the evaluator–optimizer pattern in 10.5 Agentic workflow patterns.
• Verification as the bottleneck is why coding agents work (10.8 Coding agents and computer use) and why pass@k and pass^k differ (10.9 Evaluating agents).