⛓️ Chain-of-Thought Prompting

Original Paper

What is Chain-of-Thought Prompting?

Chain-of-Thought (CoT) prompting is a technique where you ask an LLM to show its reasoning in intermediate steps before arriving at a final answer. Instead of jumping directly to the answer, the model explains its thinking.

The Paper's Key Findings (Figure 1 Recreation)

Few-Shot vs Zero-Shot CoT

Wei et al. showed two approaches:

Zero-Shot CoT: "Let's Think Step by Step"

One of the most surprising findings: simply adding "Let's think step by step" to a prompt dramatically improves accuracy, even without examples.

Prompt: "Patient: 65-year-old with chest pain, shortness of breath, and history of diabetes.
What is the most likely diagnosis?"

Response: "Likely diagnosis is a heart attack."

❌ Problem: No reasoning shown, could be wrong, hard to verify
                

Prompt: "Patient: 65-year-old with chest pain, shortness of breath, and history of diabetes.
What is the most likely diagnosis?

Let's think step by step:"

Response:
"Step 1: The patient has chest pain and shortness of breath.
These are classic symptoms of acute coronary syndrome.

Step 2: The patient has a history of diabetes.
This is a major risk factor for heart disease.

Step 3: Age is 65 (middle-aged).
This increases risk for MI.

Step 4: Considering all factors...
The most likely diagnosis is an acute myocardial infarction (heart attack).

However, we should also rule out:
- Pulmonary embolism (less likely without leg symptoms)
- Pneumonia (less likely without fever/cough)
- Aortic dissection (possible but less common)

Conclusion: Most likely = MI, but immediate ECG and troponin needed."

✓ Much better: reasoning is transparent, verifiable, more accurate
                

Why Chain-of-Thought Works

According to Wei et al. (2022), CoT works because:

• Decomposition: Breaking complex problems into steps makes them easier to solve
• Constrains reasoning: Intermediate outputs force valid reasoning, not shortcuts
• Detects errors: Readers can spot mistakes in reasoning steps
• Interpretability: We can see how the model arrived at conclusions
• Compositionality: Combining simpler steps is easier than solving all at once

Performance Improvements (From Paper)

Wei et al. showed massive improvements on complex reasoning benchmarks:

When CoT Helps (and When It Doesn't)

Implementation: CoT in Medical AI

# Simple CoT implementation with Claude API
from anthropic import Anthropic

client = Anthropic()

def medical_diagnosis_with_cot(symptoms, medical_history):
    """Diagnose using Chain-of-Thought reasoning"""

    prompt = f"""
You are an experienced clinician analyzing a patient case.

Patient Information:
- Symptoms: {symptoms}
- Medical History: {medical_history}

Please diagnose this patient. Think step by step:

Step 1: Identify the key symptoms
Step 2: Consider differential diagnoses
Step 3: Evaluate risk factors
Step 4: Consider the most likely diagnosis
Step 5: What tests would you order?
Step 6: What is your final diagnosis and reasoning?

Take your time and reason through this carefully."""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )

    return response.content[0].text


# Example usage
symptoms = "Fever 101.5F, cough for 3 days, sore throat, no difficulty breathing"
history = "65-year-old, diabetic, hypertensive, smoker"

diagnosis = medical_diagnosis_with_cot(symptoms, history)
print(diagnosis)

# Output will show step-by-step reasoning, making it auditable and trustworthy
            

Limitations and Failure Modes

Further Reading