Reasoning models and test-time compute

From chains of thought to models that think

Chain-of-thought prompting (8.4 Prompting as programming) showed that models solve harder problems when they write intermediate steps. Reasoning models make this a trained habit: before the visible answer they generate a long reasoning trace — exploring, checking, backtracking — produced by the same autoregressive machinery (7.6 Decoder-only: GPT and autoregressive generation). OpenAI's o1 (September 2024) demonstrated the approach at the frontier; DeepSeek-R1 (January 2025) published the recipe with open weights; within a year every major lab shipped reasoning modes, usually with an adjustable effort or thinking budget.

Two levers are at work, and it is worth keeping them apart:

1. Training-time RL that rewards correct final answers (and sometimes good processes), which teaches the model how to use a long trace.
2. Inference-time compute: letting the model think longer, sampling several attempts, or searching — a second scaling law alongside pretraining compute (7.9 Scaling laws and compute-optimal training).

Verifiers: outcome vs process rewards

RL needs a reward. For math with a known answer or code with unit tests, the environment itself is the verifier — RL with verifiable rewards — and the reward cannot be gamed the way a learned reward model can (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI). Where no checker exists, two learned options remain: an outcome reward model (ORM) that scores the final answer, and a process reward model (PRM) that scores each step; "Let's Verify Step by Step" (2023) showed that step-level supervision helps search and reduces false positives, at the cost of expensive labels. Practical recipes mix correctness rewards with cheap format rewards (put the answer in the required tag) and sometimes length penalties.

With GRPO (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI) — sample a group, normalize rewards, clipped policy-gradient step with a KL anchor — DeepSeek-R1-Zero trained a base model with only these signals. The trace length grew over training, and behaviors appeared that no human had written into the data: re-reading the problem, verifying intermediate results, trying alternatives, and the much-quoted "aha moment" of self-correction. The production R1 added a small supervised cold start on curated traces, a second RL stage for general helpfulness, and rejection-sampled SFT data, which is the template most open reasoning models follow.

Inference-time scaling

Given a trained reasoner, accuracy on hard problems rises roughly linearly with the logarithm of test-time compute, until the problem class saturates. Ways to spend that compute:

Intuition. Pretraining buys knowledge per parameter; test-time compute buys reliability per problem. A model can be cheap most of the time and expensive only when the question is hard — if you route well.

Distillation: small models that reason

Long reasoning traces from a strong model are excellent SFT data. R1-distilled 1.5B–70B models (fine-tuned on about 800k traces) reason far better than their base models; the s1 paper reached strong math performance from only about a thousand carefully selected examples plus budget forcing. Distillation is how reasoning spread to cheap and on-device models — and a reason fine-tuning (8.2 Fine-tuning and PEFT) matters again.

Strengths, weaknesses, costs

Reasoning models excel at mathematics, code, logic puzzles, planning, and multi-step tool use (10.3 Reasoning and planning for agents), where checking is possible and deliberation pays. They are not magic: they still hallucinate facts (thinking does not create knowledge), they can overthink simple questions, their traces are long (seconds to minutes of latency and many times the output tokens, hence cost), and the visible reasoning may not faithfully reflect the computation that produced the answer (8.11 Interpretability: what is happening inside). Evaluation is also harder: models may recognize that they are being tested ("eval awareness"), and small benchmark sets give noisy scores (8.7 Evaluating LLMs).

Using reasoning models well

• Give goals, constraints and success criteria, not step-by-step scripts; skip few-shot chain-of-thought (8.4 Prompting as programming).
• Set the budget to the task: low for routine queries, high for proofs, debugging, planning.
• Route: send easy, latency-sensitive queries to a fast model and escalate hard ones; measure the split with your eval set.
• Use reasoning for planning and verification inside agents, and cheaper models for execution (10.5 Agentic workflow patterns).
• Keep sampling at provider defaults (8.5 Decoding and sampling) and control effort through the budget.

Open questions in 2026

Does RL teach genuinely new capabilities or mostly elicit and sharpen what pretraining already contains (pass@$k$ comparisons at large $k$ fuel this debate)? How far does RL scale when rewards are rubric-based rather than verifiable? Do process rewards help beyond math? Is it safe for reasoning to be hidden or compressed, given that monitoring the trace is one of our few windows into the model's plans (8.12 Safety, risks and responsible AI)? These are live research questions; treat confident answers with suspicion.

Where this shows up later

• Reasoning models are the natural planners of agents — 10.3 Reasoning and planning for agents, and change cost/latency budgets — 10.11 Building production agents.
• Evaluation of reasoning models needs care about noise and contamination — 8.7 Evaluating LLMs.
• Faithfulness and monitorability of reasoning connect to interpretability and safety — 8.11 Interpretability: what is happening inside, 8.12 Safety, risks and responsible AI.