DSPy Prompt Optimization for Beginners: A Tiny Program, Run End to End

You have a prompt that classifies support tickets. It works on the ten tickets you tried. You tweak the wording, three old cases break, and now you're maintaining a prompt by feel. DSPy prompt optimization is one answer: write down what the step takes in and puts out, write a metric, and let a program search for the examples or instructions that score best.

This post runs the smallest honest version of that. I used DSPy 3.4.0 (the latest PyPI release as of 2026-10-09) in a clean virtualenv, with a scripted stub instead of a real model. Read the limits section before you trust any number here.

What does DSPy actually do?

Per the DSPy docs, a signature declares the input and output behaviour of a module, and the model is told what to do rather than how to prompt it. You compose modules into a program, then an optimizer compiles it against your metric and training examples. Three pieces, in order:

1. Signature. Field names and types, such as ticket -> category.
2. Module. dspy.Predict calls the model once. dspy.ChainOfThought does the same and, per the docs, adds a reasoning field automatically.
3. Optimizer. Searches for demos or instructions that raise your metric.

The prompt text is generated from those pieces. That's the trade: less hand-written prose, more written-down intent. If you want the hand-built version of improving prompts with a model, the meta-prompting workflow is the manual cousin.

How do you write a DSPy signature?

Two forms, both from the DSPy docs. Inline strings:

"question -> answer"
"sentence -> sentiment: bool"
"context: list[str], question: str -> answer: str"

And a class, where the docstring becomes the task instruction:

class Triage(dspy.Signature):
    """Classify a support ticket as billing, bug or how-to."""

    ticket: str = dspy.InputField()
    category: str = dspy.OutputField()

Use the class form as soon as you want a docstring instruction or field hints (desc=). Per the docs, output fields are required unless they declare a default, a default_factory, or a type that allows None.

How do you optimize a program against a metric?

Here is the whole experiment. It uses six tickets, a stub that answers two of them wrongly on purpose, and BootstrapFewShot. Save it as dspy_demo.py and run it in an environment with pip install dspy.

import dspy
from dspy.utils import DummyLM

rows = [
    ("My card was charged twice", "billing"),
    ("App crashes when I open settings", "bug"),
    ("How do I export my data to CSV?", "how-to"),
    ("Refund me for last month please", "billing"),
    ("Login button does nothing on Safari", "bug"),
    ("Can I change my plan mid-cycle?", "how-to"),
]
trainset = [dspy.Example(ticket=t, category=c).with_inputs("ticket") for t, c in rows]

# Scripted stub: wrong on the 3rd and 5th tickets, right on the rest.
scripted = {
    "My card was charged twice": {"category": "billing"},
    "App crashes when I open settings": {"category": "bug"},
    "How do I export my data to CSV?": {"category": "bug"},
    "Refund me for last month please": {"category": "billing"},
    "Login button does nothing on Safari": {"category": "how-to"},
    "Can I change my plan mid-cycle?": {"category": "how-to"},
}
lm = DummyLM(scripted)
dspy.configure(lm=lm)

class Triage(dspy.Signature):
    """Classify a support ticket as billing, bug or how-to."""
    ticket: str = dspy.InputField()
    category: str = dspy.OutputField()

program = dspy.Predict(Triage)

def metric(example, pred, trace=None):
    return example.category == pred.category

evaluate = dspy.Evaluate(devset=trainset, metric=metric, display_progress=False, display_table=0)
print("Score before:", evaluate(program))

optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4, max_labeled_demos=0)
compiled = optimizer.compile(program, trainset=trainset)

print("Demos after compile:", len(compiled.demos))
for d in compiled.demos:
    print("  demo:", d.ticket, "->", d.category)

Real output from my run:

2026/10/08 19:33:10 INFO dspy.evaluate.evaluate: Average Metric: 4 / 6 (66.7%)
Score before: EvaluationResult(score=66.67, results=<list of 6 results>)
Bootstrapped 4 full traces after 5 examples for up to 1 rounds, amounting to 6 attempts.
Demos after compile: 4
  demo: My card was charged twice -> billing
  demo: App crashes when I open settings -> bug
  demo: Refund me for last month please -> billing
  demo: Can I change my plan mid-cycle? -> how-to

Read what happened. The optimizer ran the program over the training set, kept only the traces where the metric said yes, and stored those four as demos on the predictor. The two tickets the stub got wrong never became demos. That filter is the heart of BootstrapFewShot: the docs describe it as generating demos with a teacher program and keeping only those that pass your metric.

What does the compiled prompt look like?

DSPy attaches the demos as earlier chat turns before your real input. I printed the messages from the last model call (the demo turns are trimmed with ellipses here, but the shape is real):

--- system
Your input fields are:
1. `ticket` (str):
Your output fields are:
1. `category` (str):
...
In adhering to this structure, your objective is: 
        Classify a support ticket as billing, bug or how-to.
--- user
[[ ## ticket ## ]]
My card was charged twice
--- assistant
[[ ## category ## ]]
billing
...
--- user
[[ ## ticket ## ]]
Can I change my plan mid-cycle?

Your docstring became the objective. Each bootstrapped demo is a user and assistant turn. Ten messages went out in total for one call: one system message, four demo pairs and the real question. Seeing this is the fastest way to stop treating DSPy as magic. It built a few-shot prompt from examples that passed your test.

Which optimizer should you pick?

The DSPy optimizers page lists these, and gives size-based guidance:

The sizing advice, paraphrased: around 10 examples, start with BootstrapFewShot; 50 or more, try BootstrapFewShotWithRandomSearch; for instruction-only optimization or longer runs with about 200 examples or more, try MIPROv2. The page says nothing about the 10 to 50 range, so don't pretend it does.

How do you run this on a real model?

Swap the stub for a real LM and keep everything else. The line below is unrun, because I had no API key in this session, and the model string is a placeholder in the provider/model format. Check the DSPy docs for the current form.

lm = dspy.LM("provider/model-name", api_key="...")  # unrun placeholder
dspy.configure(lm=lm)

Then do what the stub can't let you do: split your examples into a training set and a held-out set, compile on the first, and evaluate on the second. The score on the held-out set is the only number worth reporting.

What can this experiment not tell you?

• Quality gains. The stub answers the same way whatever the prompt contains. I added print("Score after:", evaluate(compiled)) and re-ran: it printed Average Metric: 4 / 6 (66.7%) again, identical to before. The run shows mechanics (filtering, demos, prompt shape), not improvement. I won't claim DSPy improved any accuracy figure because I measured none.
• Overfitting. Six examples is a toy. The docs' own advice to use about 200 or more for longer optimizer runs is there because optimizers can fit noise.
• Cost. A real run calls the model many times. I didn't measure token spend, so check your provider's pricing and run a small budget first.
• Metric quality. Exact string match works for labels. For free text you need a better metric, and that's where most effort goes. See building evaluation datasets and golden test sets.

When is DSPy overkill?

Skip it when:

• You have one prompt and fewer than about 20 test cases. Edit by hand, and use prompt regression tests to keep it stable.
• You can't write a metric. An optimizer with a bad metric optimizes the wrong thing, faster.
• A human must read and approve the wording, such as legal or brand copy.
• Your task changes weekly. You'll recompile constantly.

Reach for it when you have a multi-step pipeline, a metric you trust, labelled data, and a model you expect to swap out. Recompiling against a new model beats rewriting prose for it. The wider picture of prompt tooling is in the automatic prompt engineer lesson.

Checklist before your first real run

1. Write 30 or more labelled examples, and put at least 10 aside.
2. Write the metric first and test it on known good and bad outputs.
3. Run the unoptimized program and record the baseline on the held-out set.
4. Compile with the smallest optimizer for your data size.
5. Compare held-out scores. If the gain is within noise, keep the simpler prompt.