DeepEval Tutorial: Run LLM Evals in Pytest and CI Without Overspending

If you want DeepEval running in pytest today, the short version is: pip install deepeval, build an LLMTestCase per example, pick metrics, call assert_test(test_case, [metrics]) inside an ordinary test function, and run the file with deepeval test run. Some metrics need a judge model and some do not. Knowing which is which decides your bill, so this post runs both kinds and shows what each costs.

This is the tool tutorial. For the overall strategy of what to test at which layer, read the testing pyramid for LLM apps first. For Promptfoo as the config-driven alternative, see the Promptfoo guide and its CI walkthrough.

What I ran, and what I could not

I checked the docs on 2026-10-23 (getting started, metrics introduction, unit testing in CI/CD) and installed deepeval 4.3.1 on Python 3.13 in a fresh virtual environment. There were no paid API keys in this session. So:

• Run for real: ExactMatchMetric, PatternMatchMetric, a custom non-LLM metric, and GEval pointed at a stub judge that returns canned JSON.
• Not run: any judge metric against a real model. The stub proves the wiring and the number of judge calls. It says nothing about whether GEval scores are any good.
• Unrun code is labelled where it appears.

How do I install DeepEval and write the first test?

python -m venv .venv
source .venv/bin/activate
pip install deepeval pytest

The core objects, per the docs: an LLMTestCase holds one interaction (input and actual_output are required, expected_output is optional and not every metric needs it). A metric returns a score from 0 to 1 and has a threshold that decides pass or fail. assert_test runs the metrics against the case and raises if any fail.

I wrote a fake app so the tests have something to call. Replace it with a call into your real code.

# test_app.py (part 1)
from deepeval import assert_test
from deepeval.metrics import ExactMatchMetric, PatternMatchMetric
from deepeval.test_case import LLMTestCase


def fake_app(question: str) -> str:
    answers = {
        "What is the refund window?": "30 days",
        "Order id for ticket 41?": "ORD-10041",
    }
    return answers.get(question, "I don't know")


def test_exact_match():
    q = "What is the refund window?"
    case = LLMTestCase(input=q, actual_output=fake_app(q), expected_output="30 days")
    assert_test(case, [ExactMatchMetric()])


def test_order_id_pattern():
    q = "Order id for ticket 41?"
    case = LLMTestCase(input=q, actual_output=fake_app(q))
    assert_test(case, [PatternMatchMetric(pattern=r"ORD-\d{5}")])

I read the source of both metrics in the installed package. ExactMatchMetric compares the stripped strings and gives 1.0 or 0.0. PatternMatchMetric uses a regex fullmatch on the stripped output, so your pattern must cover the whole string. Neither calls a model.

Which DeepEval metrics need a judge model?

The docs say most LLM-as-a-judge metrics call an evaluation model, and name GEval as one that needs OPENAI_API_KEY or another configured LLM. In the installed package I could list many more metrics (for example AnswerRelevancyMetric, FaithfulnessMetric, HallucinationMetric, ToxicityMetric, TaskCompletionMetric). I did not open each one, so check a metric's own docs page before assuming it is free. Two I verified as non-LLM by reading the code are ExactMatchMetric and PatternMatchMetric.

The docs list a few non-LLM options and note that a metric can be built on your own scorer, so a lot of useful coverage is free. A custom metric is a small class:

# test_app.py (part 2)
from deepeval.metrics import BaseMetric


class KeywordCoverage(BaseMetric):
    def __init__(self, keywords, threshold=0.5):
        self.keywords = keywords
        self.threshold = threshold

    def measure(self, test_case, *args, **kwargs):
        text = test_case.actual_output.lower()
        hits = [k for k in self.keywords if k.lower() in text]
        self.score = len(hits) / len(self.keywords)
        self.success = self.score >= self.threshold
        self.reason = f"hit {hits} of {self.keywords}"
        return self.score

    async def a_measure(self, test_case, *args, **kwargs):
        return self.measure(test_case)

    def is_successful(self):
        return self.success

    @property
    def __name__(self):
        return "Keyword coverage"


def test_keyword_coverage():
    case = LLMTestCase(input="Summarise returns",
                       actual_output="Returns accepted within 30 days with receipt.")
    assert_test(case, [KeywordCoverage(["30 days", "receipt", "refund"])])

Keep these rule-based metrics for anything you can state as a rule: required facts present, forbidden phrases absent, length limits, IDs in the right format.

How do I plug in my own judge model?

Subclass DeepEvalBaseLLM and implement get_model_name, load_model, generate and a_generate, then pass it as model= to each metric. The docs note you must set a custom model every time you instantiate a metric. Here is a stub judge I actually ran. It returns a fixed verdict, which makes it useful for exactly one thing: proving your suite is wired correctly and counting judge calls.

# test_app.py (part 3)
import json
from deepeval.metrics import GEval
from deepeval.models import DeepEvalBaseLLM
from deepeval.test_case import SingleTurnParams


class StubJudge(DeepEvalBaseLLM):
    def __init__(self):
        self.calls = 0

    def load_model(self):
        return self

    def get_model_name(self):
        return "stub-judge"

    def generate(self, prompt: str, schema=None):
        self.calls += 1
        if schema is not None:
            return schema(**self._payload(schema))
        return json.dumps(self._payload(None))

    async def a_generate(self, prompt: str, schema=None):
        return self.generate(prompt, schema)

    def _payload(self, schema):
        names = list(schema.model_fields) if schema is not None else []
        if "steps" in names:
            return {"steps": ["Compare actual output with expected output"]}
        return {"score": 8, "reason": "stubbed verdict"}


JUDGE = StubJudge()


def test_geval_with_stub_judge():
    metric = GEval(
        name="Correctness",
        criteria="Is the actual output correct given the expected output?",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5,
        model=JUDGE,
    )
    case = LLMTestCase(input="refund window?", actual_output="30 days", expected_output="30 days")
    assert_test(case, [metric])
    print("judge calls:", JUDGE.calls)

Two things I learned from running it. First, GEval called the judge twice for one test case: once to generate evaluation steps from my criteria, and once to score. Multiply that by cases and metrics when you estimate cost. Second, my stub returned a score of 8 and the table showed 0.8, so GEval normalises the judge's 0 to 10 score onto 0 to 1 (this is what I observed with this stub, not something I read in the docs).

The command and the real output of the four tests:

DEEPEVAL_TELEMETRY_OPT_OUT=YES deepeval test run test_app.py

judge calls: 2
============================= slowest 10 durations =============================
0.03s call     test_app.py::test_geval_with_stub_judge
0.02s call     test_app.py::test_exact_match
0.02s call     test_app.py::test_order_id_pattern
0.01s call     test_app.py::test_keyword_coverage
4 passed, 4 warnings in 0.11s

Test case             Metric          Score  Status
test_exact_match      Exact Match     1.0    PASSED
test_order_id_pattern Pattern Match   1.0    PASSED
test_keyword_coverage Keyword coverage 0.67  PASSED   (hit ['30 days', 'receipt'] of [...])
test_geval_with_stub  Correctness     0.8    PASSED   (model=stub-judge, reason=stubbed verdict)

Evaluation completed (time taken: 0.9s | token cost: None)
Pass Rate: 100.0% | Passed: 4 | Failed: 0

I condensed the results table; the real one is a wider box layout. DEEPEVAL_TELEMETRY_OPT_OUT is the opt-out variable I used to keep the run offline-friendly; check the docs if you need a guarantee about telemetry.

A failing case is readable. I made the app answer "14 days" against an expected "30 days" and plain pytest reported:

AssertionError: Metrics: Exact Match (score: 0.0, threshold: 1.0, strict: False, error: None, reason: The actual and expected outputs are different.) failed.

How do I use a real judge model, including a cheap or local one?

The docs describe two routes. For Ollama, deepeval set-ollama --model=<model> configures it globally (default base URL http://localhost:11434), and deepeval unset-ollama switches back to OpenAI. For anything else, subclass DeepEvalBaseLLM as above and call your provider inside generate.

Below is an unrun sketch for an OpenAI-compatible endpoint. I did not run it because I had no key. The structure follows the stub I did run; the client calls follow the standard OpenAI Python SDK.

# UNRUN: needs a real key and model id
import os
from openai import OpenAI
from deepeval.models import DeepEvalBaseLLM


class GatewayJudge(DeepEvalBaseLLM):
    def __init__(self, model_id: str):
        self.model_id = model_id
        self.client = OpenAI(
            base_url="https://api.aicredits.in/v1",
            api_key=os.environ["JUDGE_API_KEY"],
        )

    def load_model(self):
        return self.client

    def get_model_name(self):
        return self.model_id

    def generate(self, prompt: str, schema=None):
        resp = self.client.chat.completions.create(
            model=self.model_id,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            response_format={"type": "json_object"} if schema else None,
        )
        text = resp.choices[0].message.content
        return schema.model_validate_json(text) if schema else text

    async def a_generate(self, prompt: str, schema=None):
        return self.generate(prompt, schema)

The docs carry a warning worth repeating: DeepEval cannot guarantee evaluations work as expected with a custom model, and weaker models may return invalid JSON. So a cheap judge saves money and can break the run. Check a sample of its verdicts against human labels first; the LLM-as-judge pitfalls post has a script for that, and the Cohen's kappa post shows the agreement maths.

Disclosure: the AICredits link below is an affiliate link. AICredits is an OpenAI-compatible gateway (INR via UPI from Rs 50), and a budget-capped key is a handy way to make sure a runaway eval loop cannot spend more than you topped up. Non-affiliate ways to get the same cap: your provider's own project spend limits, a self-hosted LiteLLM proxy with per-key budgets, or a local model through Ollama.

How do I run DeepEval in CI?

The docs' GitHub Actions example installs dependencies, sets OPENAI_API_KEY (the judge) and optionally CONFIDENT_API_KEY (to send results to Confident AI), and runs deepeval test run test_llm_app.py. Here is a minimal version of that idea adapted to pip. Unrun, since I have no Actions runner here.

# .github/workflows/evals.yml (UNRUN)
name: llm-evals
on:
  pull_request:
    paths: ["prompts/**", "app/**", "tests/evals/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Cheap rule-based evals
        run: deepeval test run tests/evals/test_rules.py
      - name: Judge evals (capped key)
        if: github.event.pull_request.draft == false
        env:
          JUDGE_API_KEY: ${{ secrets.JUDGE_API_KEY }}
        run: deepeval test run tests/evals/test_judge.py -i -c

The deepeval test run --help output from 4.3.1 lists these options, which I ran on 2026-10-23: -n (processes), -r (repeat), -c (use cache), -i (ignore errors), -s (skip cases with missing parameters), -d (display all, failing or passing) and -x (exit on first failure). The docs describe -i as ignoring errors and -c as using the local cache so partially successful runs do not re-run errored cases.

For near-threshold tests, the installed version supports flaky=True on a test case (a failure then raises a warning instead of an exception, per the source I read and the docs). Use it sparingly. A flaky flag that spreads is how a suite stops meaning anything.

How do I control judge-call cost?

Judge calls scale as cases times metrics times calls per metric. Use this order:

1. Split the suites. Rule-based tests (free) run on every push. Judge tests run on non-draft pull requests or nightly.
2. Sample. Run 20 to 30 cases per PR and the full set nightly. Choose the sample size by how much noise you can tolerate; it is a trade-off, not a fact.
3. Count calls first. My GEval run made two calls per case. Run your suite once with a counting wrapper like the stub above before you point it at a paid key.
4. Cache. Use -c so unchanged cases are not judged again.
5. Use the smallest judge that agrees with your humans. Test that with labels, not with a leaderboard.
6. Cap the key. A key with a hard budget turns a bug into a failed job instead of a bill.

Should you pick DeepEval?

Pick it if your team lives in pytest and wants evals to look like tests, with a rich metric library for RAG and agents. Skip it, or use it only for the rule-based parts, if you want evals defined in YAML next to prompts (Promptfoo is closer to that), or if you are not ready to maintain judge metrics. Either way, the cheap layers do most of the work, and the judge is the part you validate hardest.