Build a Golden Dataset for LLM Evals From Production Logs

You have 40,000 chat logs and no idea which 80 of them should be in your eval set. Taking the first 80, or a random 80, gives you a golden dataset that mostly tests "where is my order" over and over. The set you want is small, deduplicated, free of personal data, tilted toward the cases that went wrong, and split so the holdout can't be gamed by accident.

This post builds that pipeline in about 130 lines of standard-library Python. I ran it on 600 invented support-bot logs generated with a seeded script (not real customer data, and the numbers below describe my fake logs, not your traffic). You'll get the script, the real output, and the labelling rules that make the result usable. If you want the general theory of eval sets first, read building evaluation datasets and golden test sets. This post covers the part that one skips: getting from raw production logs to a labelling queue.

What should a golden dataset made from production logs contain?

Four kinds of rows, in roughly this order of importance:

1. Failures. Conversations with a thumbs-down, an escalation to a human, a retry, or an angry follow-up. These are where your next regression will hide.
2. Common intents. The top few things users ask, so a change that breaks refunds can't ship unnoticed.
3. Odd shapes. Very short ("refund??"), very long multi-issue messages, mixed languages, pasted error text.
4. Attacks and out-of-scope asks. Someone asking the bot to ignore its instructions is a real production input. It belongs in the set with an expected behaviour of "decline".

Each row needs the input, a bucket, the reason it was picked, and a label that says what a good response does. Notice that this isn't "the exact answer". For most LLM features you label behaviour: which facts must appear, which things must not, whether it should escalate.

How do I clean production logs before anyone labels them?

Do the cleaning in code, in a fixed order, and print the row count after each step so you can see what each stage removes.

Scrub before anything else. The labelling queue is a new copy of your users' words, usually opened by more people than the production database. Replace emails, phone numbers and order IDs with placeholders first. Regexes catch the obvious cases; they miss names, addresses and free-text details ("my daughter Aanya's school"). For a production pipeline use a proper detector, as in PII redaction before LLM calls with Presidio and regex, and have a person skim the sampled rows before they go to labellers. Retention and consent rules for logs are a legal question: see the DPDP Act checklist for AI app developers and ask a professional.

Normalise, then dedupe exactly. After scrubbing, "where is my order #48213" and "where is my order #90017" are the same row. Lowercase, collapse whitespace, strip filler like "pls", then keep one row per normalised string and remember how many collapsed into it. That count is useful: it's your estimate of how common the intent is.

Dedupe near-duplicates. Typos and openers ("hi, ", "urgent: ") defeat exact matching. Character 3-gram Jaccard similarity at 0.8 is cheap, needs no embeddings and works at this scale. It's a quadratic loop and I only ran it on a few hundred rows, so time it on your data; if it crawls, switch to MinHash or an embedding index.

Bucket. Real logs rarely carry a clean intent label. A keyword router is crude, but good enough to stop one intent dominating. Anything over 140 characters goes to a long_multi_issue bucket because those are the messages that break prompts.

Sample on purpose. Half the budget goes to hard cases (thumbs-down or escalated). The other half is spread round-robin across buckets, so a rare bucket isn't drowned by the common one.

Split by user. Hash the user ID to assign dev or holdout. Never split by row.

The script

Save as build_golden.py. It reads JSONL with the fields id, user_id, input, thumbs and escalated, and writes label_queue.csv.

"""Turn raw production logs into a labelling queue for a golden dataset.
Usage: python3 build_golden.py logs.jsonl out_dir [target_size]
"""
import csv, hashlib, json, os, random, re, sys
from collections import Counter, defaultdict

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
PHONE = re.compile(r"\+?\d[\d\s-]{8,}\d")
ORDER = re.compile(r"#\d{4,}")


def scrub(text):
    text = EMAIL.sub("<EMAIL>", text)
    text = PHONE.sub("<PHONE>", text)
    return ORDER.sub("<ORDER>", text)


def normalise(text):
    text = re.sub(r"\s+", " ", text.strip().lower())
    text = re.sub(r"\b(pls|please)\b", "", text)
    return re.sub(r"\s+", " ", text).strip(" ?.!")


def shingles(s, k=3):
    return {s[i:i + k] for i in range(max(1, len(s) - k + 1))}


def jaccard(a, b):
    return len(a & b) / len(a | b) if a | b else 1.0


BUCKETS = [
    ("refund", ("refund", "money back", "return")),
    ("shipping", ("where is", "arrive", "tracking", "parcel", "late")),
    ("account", ("log in", "login", "password")),
    ("billing", ("charged", "card", "invoice")),
]


def bucket(norm):
    if len(norm) > 140:
        return "long_multi_issue"
    for name, keys in BUCKETS:
        if any(k in norm for k in keys):
            return name
    return "other"


def main(path, out_dir, target):
    os.makedirs(out_dir, exist_ok=True)
    rows = [json.loads(line) for line in open(path)]
    print("raw rows", len(rows))
    for r in rows:
        r["clean"] = scrub(r["input"])
        r["norm"] = normalise(r["clean"])

    seen = {}                                   # exact dedupe, keep a count
    for r in rows:
        if r["norm"] in seen:
            seen[r["norm"]]["dupe_count"] += 1
        else:
            r["dupe_count"] = 1
            seen[r["norm"]] = r
    rows = list(seen.values())
    print("after exact dedupe", len(rows))

    kept, kept_sh = [], []                      # near dedupe, greedy
    for r in rows:
        sh = shingles(r["norm"])
        hit = next((k for k, ks in zip(kept, kept_sh) if jaccard(sh, ks) >= 0.8), None)
        if hit:
            hit["dupe_count"] += r["dupe_count"]
            hit["thumbs_down_any"] = hit.get("thumbs_down_any", False) or r["thumbs"] == "down"
        else:
            r["thumbs_down_any"] = r["thumbs"] == "down"
            kept.append(r)
            kept_sh.append(sh)
    rows = kept
    print("after near dedupe", len(rows))

    for r in rows:
        r["bucket"] = bucket(r["norm"])
        r["hard"] = bool(r["thumbs_down_any"] or r["escalated"])

    rng = random.Random(42)                     # seeded: the sample is reproducible
    hard = [r for r in rows if r["hard"]]
    easy = [r for r in rows if not r["hard"]]
    rng.shuffle(hard)
    rng.shuffle(easy)
    picked = hard[: target // 2]
    by_bucket = defaultdict(list)
    for r in easy:
        by_bucket[r["bucket"]].append(r)
    names, need, i = sorted(by_bucket), target - len(picked), 0
    while need > 0 and any(by_bucket.values()):
        b = names[i % len(names)]
        if by_bucket[b]:
            picked.append(by_bucket[b].pop())
            need -= 1
        i += 1

    for r in picked:                            # split by user, never by row
        h = int(hashlib.sha256(r["user_id"].encode()).hexdigest(), 16) % 100
        r["split"] = "holdout" if h < 30 else "dev"

    fields = ["id", "split", "bucket", "hard", "dupe_count", "clean",
              "expected_behaviour", "must_include", "must_not_include", "labeller", "notes"]
    with open(os.path.join(out_dir, "label_queue.csv"), "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
        w.writeheader()
        for r in picked:
            w.writerow({**r, "expected_behaviour": "", "must_include": "",
                        "must_not_include": "", "labeller": "", "notes": ""})

    print("sampled", len(picked), dict(Counter(r["split"] for r in picked)))
    both = {r["user_id"] for r in picked if r["split"] == "dev"} & \
           {r["user_id"] for r in picked if r["split"] == "holdout"}
    print("users in both splits:", len(both))


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 else 60)

What did it do to 600 logs?

I generated 600 invented support-bot rows (six intents, typos, openers, emails, phone numbers, order IDs, a 20 percent rate of thumbs-down or escalated rows) and ran the full version of the script above with a target of 60. The full version I ran also printed a per-bucket table; the output:

Funnel
  raw rows                               600
  after exact dedupe                     498
  after near dedupe (jaccard>=0.8)       419
  unique rows kept                       419
  sampled for labelling                   60
Sample by bucket x hard
  account           easy     6
  account           hard     3
  billing           easy     6
  billing           hard     1
  long_multi_issue  easy     1
  other             easy     6
  other             hard     5
  refund            easy     6
  refund            hard    16
  shipping          easy     5
  shipping          hard     5
Split: {'holdout': 17, 'dev': 43}
Users in both splits: 0
Rows still containing email/long digits: 0

Three things to read from that.

Dedupe removed 30 percent of the rows (600 to 419), and that's with synthetic data built from a handful of templates. Real logs will vary, so measure yours before assuming a figure.

Random sampling would have under-represented the failures. 121 of my 600 rows were hard (thumbs-down or escalated). I drew 1,000 random samples of 60 and counted hard rows in each: the minimum was 2, the mean 12.0 and the maximum 21. The stratified sample has 30 by construction. A single random draw could easily have left you with a set that has two failures in it.

The long multi-issue bucket is nearly empty (1 row). That's a finding, not a bug: after dedupe there were few of them. If those messages matter to you, write extra rows by hand or pull them from a longer time window.

The split is also worth checking: 17 holdout and 43 dev rows is 28 percent holdout against the 30 percent I asked for, because the hash splits users, not rows. Zero users appear on both sides.

How do I label the queue so the labels are usable?

The CSV has empty expected_behaviour, must_include and must_not_include columns. Fill them with behaviour, not prose.

Rules that keep the set honest:

• Write the label before you look at the model's answer. Otherwise you'll label whatever it said as correct.
• Two people label 20 rows, then compare. If you disagree on more than a couple, your rubric is vague. Fix the rubric, not the rows.
• Keep the hard and dupe_count columns. Report results per bucket, and weigh by dupe_count if you want a number that reflects real traffic.
• Record why a row is in the set. "Added after the 2026-10-12 refund incident" saves you from deleting it a year later.

If you want a model to draft labels, this is an untested template; I didn't run it on a live model:

You are helping label an evaluation set for a customer-support bot.
Policy: <paste your policy here>.
For the user message below, write: (1) expected_behaviour in one sentence,
(2) must_include as a short list, (3) must_not_include as a short list.
Do not write the full reply. If the message is ambiguous, say what clarifying
question the bot should ask.

User message:
<message>{{CLEAN_TEXT}}</message>

A person then edits every draft. Grading a model against labels that a model wrote is circular; see LLM-as-judge pitfalls for how that bias creeps in.

How do I keep the dataset useful after the first week?

• Add every incident. Each production failure becomes a row, with a note. The set should grow from your mistakes.
• Refresh the hard slice monthly, and keep a frozen core so scores stay comparable across months.
• Never tune on the holdout. If you read holdout failures to rewrite a prompt, that row has become dev data. Move it, and be honest that the holdout shrank.
• Wire it to CI. A golden set nobody runs is a spreadsheet. Prompt regression tests in CI with Promptfoo shows the plumbing.

What this post didn't verify

The script handles invented data. Real logs have multi-turn conversations, attachments, several languages and messy encodings, none of which I tested. The near-dedupe threshold of 0.8 is a starting value I picked, not a tuned one; look at a sample of what it merged before trusting it. The regex scrubber is deliberately basic and will miss names and addresses.