Building Golden Evaluation Datasets from Production Traces

I used to run evals on whatever dataset I had around. It was noisy, and I kept digging into regressions that turned out to be bad data. What fixed it was a small, versioned "golden dataset" built from real traces, which I keep stable.

A golden dataset is a stable, representative set of real examples that you rerun after every prompt or model change. If your traces are already in LangSmith, you can build one from those runs.

What a stable eval set buys you

Production behavior shifts even faster than your prompts and models do. Without a stable eval set, you usually learn about regressions from frustrated users. A golden set helps by giving you:

• A repeatable benchmark you trust.
• A baseline for prompt and model comparisons.
• Regressions caught early, in your own eval runs.
• A small set you can run frequently (daily or in CI).

The examples come from real traces, so they are questions users actually asked.

Example: licensing assistant in production

One setup I have worked on is a LangGraph assistant that helps sales reps quote and manage software licenses for Microsoft and Bitdefender. Recommendations start from the product database, but selection and adaptation happen in the LLM/agent. Users ask for bundle recommendations, renewal terms, and region-specific compliance. A wrong SKU or term can kill a deal. Requests like these came up often.

In that setting, a golden dataset built from real traces helps catch regressions like:

• A 2-year renewal is shortened to 1 year.
• A region-restricted SKU is being suggested outside its allowed market.
• A missing compliance clause in the generated quote summary.

What makes a dataset "golden"

A golden dataset has three properties that a random dump of traces lacks:

1. Representative

   • It covers your top user intents.
   • It includes failure-prone queries and edge cases.

2. Labeled

   • It has reference answers or scoring rubrics (exact SKUs, correct terms, required clauses).
   • It is structured enough for automated evaluators.

3. Stable and versioned

   • You can rerun it across weeks and compare results.
   • You know exactly when and why it changed.

If you only have one dataset in LangSmith, make that one your golden set.

Step 1: Harvest candidate traces

Start by pulling recent traces from the flows that matter most (your main product paths). Use a fixed time window (for example, the last 7 days) so you are comparing the same traffic conditions over time. I usually stick to a normal week unless there was a big launch or outage.

If you need a quick refresher on datasets and runs, the official LangSmith docs are a good reference.

Pseudo-code example:

# Pseudo-code: harvest traces and create a golden dataset
runs = langsmith.list_runs(project_name="licensing-assistant", since_days=7)

# Stratified sampling keeps intent/region biases in check
candidates = [
    r for r in runs
    if r.metadata.get("intent") in {"quote", "renewal"}
    and r.metadata.get("region") in {"us", "eu"}
]
# Pseudo: stratify so you don't overfit to one slice (intent x region)
buckets = group_by(candidates, key=lambda r: (r.metadata.get("intent"), r.metadata.get("region")))
candidates = sample_evenly(buckets, n_per_bucket=30)  # ~120 total

dataset = langsmith.create_dataset(
    name="golden-licensing-v1",
    schema={
        "input": "string",
        "expected": "object",
        "metadata": "object"
    }
)

for run in candidates:
    user_input = run.inputs.get("query") or run.inputs.get("input") or ""
    user_input = redact_pii(user_input)
    # Seed expected from outputs, then review/edit in LangSmith to make it correct.
    # NOTE: expected values must become ground truth (don’t trust model output).
    dataset.add_example(
        input=user_input,
        expected={
            "sku": run.outputs.get("sku"),
            "term_months": run.outputs.get("term_months"),
            "region": run.outputs.get("region"),
            "compliance_clauses": run.outputs.get("compliance_clauses", []),
            "price_breakdown": run.outputs.get("price_breakdown", {})
        },
        metadata={
            "intent": run.metadata.get("intent"),
            "region": run.metadata.get("region"),
            "product_line": run.metadata.get("product_line"),
            "catalog_version": run.metadata.get("catalog_version"),
            "pricing_version": run.metadata.get("pricing_version"),
            "compliance_version": run.metadata.get("compliance_version")
        }
    )

My quick selection rules:

• Include common, boring queries. They are your baseline.
• Include "spiky" queries (long or multi-step). Adversarial queries belong here too.
• Remove any runs with clear input errors or missing data.
• Add negative cases intentionally (cross-region requests, expired SKUs, missing customer profiles).
• Redact and normalize PII before saving runs (customer names, tenant IDs, emails).

If you have multiple pipelines, create a candidate set per pipeline and then merge for the golden set.

Step 2: Curate for coverage

Curation is where you decide what goes in. A noisy example you let through now can show up later as a regression that is not real.

Practical checklist:

• Coverage by topic or intent (use tags).
• A balance of easy vs hard cases.
• Known historical failures (regressions you already saw).

Aim for 50-200 examples for your first golden set. I usually start with 100 because it is small enough to review by hand.

If you can, split your datasets into a "golden" (hand-verified) set and a larger "silver" set (auto-harvested). Run evals on the golden set and use silver to discover new patterns.

Step 3: Add labels and grading criteria

Without labels, you are just replaying inputs. You need at least one of:

• Reference outputs (expected answers).
• Grading rubrics (criteria for an evaluator).
• Metadata needed for a downstream evaluator.

If full reference answers are too expensive, start with a rubric and an LLM-as-judge evaluator. For detecting regressions, that is enough. For strict checks, prefer structured expected outputs (JSON targets) so you can validate fields like sku, term_months, and region directly instead of comparing free-form text.

Step 4: Enforce a schema

Schema is how you prevent bad examples from sneaking into your golden set. A minimal schema typically includes:

• Input (the user query or input payload).
• Reference output (optional but ideal).
• Metadata (topic, difficulty, failure mode).

Even if you do nothing else, add a schema to your dataset. It cuts down on flaky evals caused by malformed examples.

Step 5: Tag examples for slicing

Tags show a regression in one slice even when the whole-set average does not reveal it. You can slice results by:

• Topic or intent.
• Difficulty (easy, medium, hard).
• Failure mode (hallucination, retrieval miss, formatting).

Start with 3-5 tags. I once over-tagged a dataset and never used half the tags again.

Step 6: Run evals and compare variants

Use the dataset for:

• Prompt A vs Prompt B comparisons.
• Model A vs Model B comparisons.
• Release vs previous release regressions.

Keep your experiment design consistent: same dataset version, same evaluator setup, and a fixed number of repetitions if you need variance control.

Track metrics over time and keep a log of the regressions you find.

Step 7: Maintain the golden set

A golden dataset still needs updates. Keep them small and deliberate:

• Add new user intents when they become important.
• Rotate in new edge cases every few weeks.
• Keep a core set that never changes.

When you update the dataset, version it and record why it changed. I keep it simple with names like golden-v1, golden-v2, plus a tiny changelog.

The flow is:

Prod traces -> filter and scrub -> label and tag -> golden dataset -> eval runs -> dashboard

Suggested metrics

Pick metrics that match your product. I try to keep it to two or three:

• Correctness or rubric score.
• Retrieval relevance score (if RAG).
• Latency and cost (for operational changes).

Avoid overfitting to a single metric. The golden set is there to find examples that passed before and fail now.

Proof example (before and after)

In the licensing assistant, the first golden set had 100 examples, and 12% of them failed schema validation due to missing region metadata, among other issues like term and SKU mismatches. After enforcing schema and adding a default region mapping, invalid rows dropped to 2% of those 100 examples and eval reruns stopped flaking.

Common mistakes to avoid

• Using only "easy" examples.
• Skipping schema validation.
• Never refreshing the dataset.
• Changing the dataset while comparing experiments.
• Storing raw PII without redaction or normalization.

Before your next prompt change

If you want to start this week, here is a simple checklist:

• Pick 100 traces.
• Tag them with 3-5 categories.
• Add a schema and one evaluator.
• Rerun after your next prompt change.