Online Evaluation Guardrails: Catching LLM Drift Before Users Do

Passing offline evals doesn't keep a system correct once it's in production. Its behavior keeps changing, and online evals are how you catch that drift before anyone reports it.

My setup for the offer-bundling assistant runs small checks on a sample of live traffic and alerts when a failure rate crosses a threshold.

Offline vs online evals

You need both:

• Offline evals run on known examples and stop regressions before a release.
• Online evals watch real traffic in production and pick up failure modes that no known example covered.

Where to add guardrails

In the offer-bundling flow, I add checks at:

• Output validation (schema conformity)
• Compliance clause coverage
• Budget threshold

Each is a small evaluator with a simple rule.

A minimal online evaluator

One of the checks looks for missing compliance clauses in the EU.

# Pseudo-code: online guardrail
if request.region == "eu":
    if "gdpr" not in output.compliance_clauses:
        record_failure("missing_gdpr")

The rule only reads the request region and the output's compliance clauses, so running it is cheap.

Thresholds and alerts

I set thresholds based on impact. Example:

• If more than 2% of EU bundles miss GDPR, alert.
• If more than 5% exceed budget, alert.

I set both at the point where users would start to feel the problem, so smaller anomalies do not raise an alert.

Sampling to control cost

You do not need to evaluate 100% of traffic. Start with:

• 5% sampling for low-risk checks
• 20% for high-risk checks

Handling false positives

Most alerts end up being harmless, so I keep a short review loop:

• Sample a few failures daily.
• Confirm if they are real.
• Adjust thresholds or rules.

How this connects to the project

The offer-bundling assistant already has schema validation and golden datasets. Online guardrails come after both, and they do not replace a test suite. They cover drift and the real-world edge cases that never showed up in tests.