The fastest way I've found to debug an LLM app isn't logs. It's a trace. A good trace shows exactly where the chain broke, which inputs triggered it, and what the model returned.
When this assistant started serving real users, tracing became the difference between guessing and fixing incidents. When something breaks in production, I don’t want a theory. I want a trace.
This post is the 10-minute workflow I use on the offer-bundling assistant project. It's small, but the steps apply to any pipeline: reproduce, locate, compare, fix.
In 30 seconds
- Traces beat logs because they show the full run tree and inputs/outputs.
- Workflow: reproduce -> locate the failing node -> compare good/bad runs -> fix.
- Use tags and metadata so you can slice failures fast.
Key takeaways
- One trace is worth a dozen log lines.
- Compare two runs to isolate the root cause.
- Keep a short debugging checklist for future incidents.
Why traces beat logs
Logs give you strings. Traces give you structure: a tree of runs with inputs, outputs, latency, and metadata. When a bundle recommendation is wrong, it affects pricing, compliance, or both, I do not want to guess which step failed. I want to see it.
In the offer-bundling project, a single request can touch:
- catalog lookup
- prompt assembly
- model inference
- output validation
A trace shows all of that in one place.
Minimal setup (what you actually need)
You need only two things:
- LangSmith API key
- One place where your app sends runs
In the demo project, I set these in .env so every run is traced automatically.
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=langsmith-offer-bundlesTrace anatomy in 60 seconds
Every trace has:
- A root run (the overall request)
- Child runs (each step or node)
- Inputs, outputs, and metadata
Think of it like a stack trace for LLM workflows.
The 10-minute debugging workflow
Here is the exact loop I follow:
- Reproduce
- Run the same request with the same inputs.
- Tag the run (for example:
issue:bundle-missing-sku).
- Locate
- Open the trace tree.
- Find the node where the output changes or fails validation.
- Compare
- Grab a good run and a bad run.
- Compare inputs and outputs side by side.
- Fix
- Fix the prompt, schema, or data issue.
- Rerun the same input to confirm.
This takes minutes if your traces are clean.
What failures look like in traces
Common root causes in the offer-bundling flow:
- Missing field in the model output (schema mismatch)
- Wrong region selected due to metadata drift
- Over-budget recommendation because price fields were missing
In each case, the trace shows the exact step where the wrong value appeared.
A simple checklist I keep
- Can I reproduce the failure with the same input?
- Which node produced the wrong output?
- Is the output invalid, or just wrong content?
- Did a prompt or schema change recently?
- Can I compare a good run vs a bad run?
Diagram idea
User request -> catalog lookup -> prompt -> model -> validator -> result
Closing thought
This workflow applies whether you’re using LangSmith, custom tracing, or internal observability tooling.
If you're still debugging LLM workflows by reading logs, you're wasting time. Add tracing once and you will stop guessing. The root cause will usually show up in two traces: one good, one bad.