Chain of Thought (CoT) prompting is one of the most powerful techniques in a prompt engineer's toolkit. It's the difference between asking an AI to guess an answer and asking it to work through the problem.
What is Chain of Thought Prompting?
Chain of Thought prompting is a technique where you instruct the AI to reason step-by-step before giving a final answer. Instead of jumping directly to a conclusion, the model shows its work.
Without CoT:
Prompt: A store has 24 apples. They sell 1/3 in the morning and 1/4 of the remainder in the afternoon. How many are left?
Response: 14 apples (wrong)
With CoT:
Prompt: A store has 24 apples. They sell 1/3 in the morning and 1/4 of the remainder in the afternoon. How many are left? Think step by step.
Response:
- Start: 24 apples
- Sell 1/3 in morning: 24 × (1/3) = 8 sold → 16 remaining
- Sell 1/4 of remainder: 16 × (1/4) = 4 sold → 12 remaining
Answer: 12 apples (correct, and verifiable)
2026 update: reasoning models do this for you. GPT-6 Astra, Claude Fable 5.1 and Opus 5.5, and Gemini 3.x all reason in a hidden phase before answering. For those models, drop "think step by step" and the
<thinking>tags below. Give the task, constraints and output format, and control depth with the effort setting. The techniques in this guide still apply to fast, small and open-weight models. The updated CoT lesson covers both cases side by side.
Why Does It Work?
Large language models predict the next token. When forced to reason out loud, each step becomes context for the next — meaning the model builds on correct intermediate steps rather than pattern-matching to a potentially wrong answer.
Research shows CoT significantly improves performance on:
- Math word problems
- Multi-step logic puzzles
- Planning tasks
- Code debugging
- Any task requiring sequential reasoning
Three Levels of Chain of Thought
Level 1: Basic — "Think step by step"
The simplest implementation. Just add a phrase that signals you want reasoning.
What is 17% of 250? Think step by step.
Level 2: Guided CoT — Tell it the steps
Provide a framework for how the model should reason.
You are a financial analyst reviewing a Q4 report.
Here is the report: [paste report]
Analyze this by:
1. Identify the key revenue metrics and YoY changes
2. Assess the cost structure and margins
3. Identify 2-3 risks mentioned
4. Give a one-sentence investment outlook
Be thorough in each step before moving to the next.
Level 3: Structured CoT with XML tags
For models without built-in reasoning, separate the reasoning from the answer so your code can strip it.
You are a legal assistant reviewing a contract clause.
<clause>
The licensor may terminate this agreement with 30 days written notice if licensee
fails to cure any material breach within 15 business days of receiving notice.
</clause>
First think through the implications in a <thinking> block,
then provide your final analysis in an <analysis> block.
The <thinking> tag gives the model dedicated space to reason without polluting the final output. Skip this on reasoning models: they already thought in a hidden channel, and a visible <thinking> block just adds billed output. Ask for a short rationale instead.
When to Use Chain of Thought
Use CoT when:
- The task involves multiple steps
- There's a right answer that can be reasoned to
- You need to verify the logic, not just the output
- The task involves math, analysis, or planning
Don't use CoT when:
- The task is simple and direct (translating a phrase, writing a subject line)
- Speed and low token count matter more than perfect reasoning
- The task is creative rather than logical
Key CoT Phrases to Use
Add any of these to activate step-by-step reasoning:
- "Think step by step."
- "Let's work through this carefully."
- "Reason through this before giving your answer."
- "Walk me through your reasoning."
- For XML: use
<thinking>tags (models without built-in reasoning only)
Key Takeaway
Chain of Thought prompting costs a few extra words and can dramatically improve accuracy on models that don't reason on their own. On current reasoning models the same idea is built in, so your job shifts to a precise task, clear constraints and the right effort setting. See how to prompt reasoning models for that side.
