A production LLM feature fails differently from a normal service. Nothing throws; latency is fine; the dashboard is green; and the answer is wrong. LLM observability exists because the failure signal moved from the infrastructure layer into the content itself - and none of your existing monitoring looks there.
The teams operating LLM apps well in 2026 treat them like distributed systems with an extra dimension: alongside request rates and error budgets, they trace every generation, account for every token, and continuously score output quality. The tooling has matured - Langfuse, Arize Phoenix, Braintrust, and the OpenTelemetry GenAI semantic conventions cover most needs - but the discipline is what separates operable systems from black boxes.
Traces Are the Unit of Debugging
A single user interaction in a modern LLM app fans out into retrieval calls, tool invocations, guardrail checks, and one or more model generations. When the answer is wrong, the question is always what the model actually saw - and only a trace answers it.
- ▸Span everything in the chain: Each retrieval, rerank, tool call, and generation is a span carrying its inputs, outputs, model version, and token counts. The OpenTelemetry GenAI conventions give you a vendor-neutral schema, so instrument once and keep your backend options open.
- ▸Capture the rendered prompt, not the template: Bugs live in the assembly - a truncated context, a stale variable, retrieved chunks in the wrong order. Store what was actually sent.
- ▸Redact before you persist: Traces are now a store of user data and retrieved documents. PII scrubbing, retention limits, and access controls on the trace store are prerequisites, not enhancements - your DPO should know it exists.
- ▸Link feedback to traces: A thumbs-down that cannot be joined to the exact trace that produced it is nearly worthless; the join is the whole point.
The Metrics That Page Someone
Token spend and latency are the operational vitals. Track cost per feature and per tenant, not just per API key - one customer's pathological usage pattern can quietly distort your bill. Watch p95 and p99 time-to-first-token separately from total generation time, because they degrade for different reasons and users perceive them differently. And alert on structural signals that precede quality complaints: refusal-rate spikes, output-length collapse, tool-call error rates, retrieval returning empty. These are cheap to compute and catch provider-side changes hours before your users file tickets.
Online Evaluation Without Fooling Yourself
Offline golden sets guard deployment; online evaluation tells you what production actually looks like. The workhorse is LLM-as-judge - a second model scoring sampled outputs for groundedness, relevance, and policy compliance - and it is genuinely useful right up until teams start trusting it blindly.
1. Sample, do not score everything: a few percent of traffic, stratified by feature and tenant, keeps cost sane and coverage honest. 2. Calibrate the judge against humans: score a few hundred cases both ways, measure agreement, and re-check whenever you change the judge model or its rubric. 3. Pin the judge model version: an unpinned judge silently redefines your quality metric when the provider upgrades it. 4. Trend scores by cohort: a stable global average can hide one language or one document type falling off a cliff. 5. Route the worst outputs to human review: the tail is where incidents live, and it is also your best source of new golden-set cases.
Drift Comes From Three Directions
LLM systems drift even when you deploy nothing. Providers retire and revise hosted models - version pinning narrows this, but deprecation deadlines force migrations on their schedule, not yours. Your data drifts: the document corpus grows, user intent shifts, new products change the vocabulary. And your own system drifts through the accumulation of small prompt and tool edits that each passed review but compound into behavior nobody designed. The defense is the same for all three: continuous evaluation against a maintained golden set, plus trace-level diffing when metrics move - compare this week's traces for a recurring query against last month's, and the divergence point is usually visible in minutes.
Observability as the License to Ship Faster
The business value of this stack is not the dashboards - it is the confidence to change things. A team with traces, calibrated online evals, and drift alarms can swap a model, rewrite a prompt, or upgrade a retrieval pipeline and know within hours whether production got better or worse. A team without them ships slowly and by argument. As AI features become the product rather than a garnish, that iteration speed compounds into a real competitive gap - and when a customer or regulator asks why the system said what it said, the trace is the difference between an answer and a shrug.