In the overall score, our bundle recommender seemed stable. One region-specific slice, though, was failing badly, and tags are what made it visible. After tracing individual failures to root cause, I saw that the aggregate metrics were the bigger problem, because they hid where users were actually failing.
The tagging in this post comes from the offer-bundling assistant. It uses a small taxonomy with consistent names, and each slice lines up with a real risk for users. I use bundles here, but the same issue can appear wherever aggregate scores hide a local failure.
In 30 seconds
- Tag examples and runs, then slice evals by topic and risk.
- A first taxonomy needs only 3 to 5 tags.
- Slicing by region and failure mode is what caught our GDPR failure.
- Compare model or prompt variants on the same tagged slices.
What the overall score misses
If you only look at overall scores, you miss where users actually struggle. Tags let you answer:
- Does the EU region fail more often?
- Are multi-step queries worse than single-step?
- Which prompts, if any, make compliance clauses drop?
A simple tagging taxonomy
Start with three dimensions:
- topic:
quote,renewal,bundle - difficulty:
easy,medium,hard - failure-mode:
missing-clause,wrong-sku,over-budget
Region (for example eu) and intent are tagged too; they come from request metadata.
Where to attach tags
In LangSmith, tags can live on:
- runs
- eval runs
Dataset examples carry the same labels as metadata, which you can filter on. In the offer-bundling assistant, I label examples by region and intent in their metadata, and I tag runs by prompt version.
Using tags in practice
Once tags exist, you can filter dashboards by tag and compare prompt versions on the same slice. They also let you isolate a regression to one slice when the average looks fine.
The eu + missing-clause slice, for example, caught a GDPR failure that was invisible in the global score.
Left unnoticed, these slice-level failures would have shipped incorrect or non-compliant recommendations.
Building tagged sets from traces
When you harvest traces for a golden dataset, auto-tag by metadata (region, intent) first, then manually review the high-risk cases.
This keeps the golden set small.
Maintain a core tagged set
I keep 30 to 50 examples tagged by the highest-risk categories and run them on every prompt change. I haven't found a quicker regression check.
Picking the first three tags
Pick three tags and apply them the same way on every run. Then compare each slice with the average.