Evidently AI · 5:08 · Watch on YouTube
Educational video on evaluating LLM applications after they ship. It covers what you can track in production (product metrics, manual log review, software monitoring, and AI quality monitoring), draws the line between monitoring and observability, and then walks through the three-step production observability workflow: tracing, online quality evaluations, and alerting. It closes by explaining why offline test datasets and regression testing still matter even once real production data exists.
Topics
- From experiments to production — why production evaluation is both an opportunity and a safety need
- What to track in production — product metrics, manual labeling, software monitoring, and LLM quality monitoring
- Monitoring vs. observability — the distinction, and the three-step production workflow
- Tracing — capturing full sessions, intermediate steps, and why product teams need access
- Online quality evaluations — delayed/sampled evals, quality metrics, dashboards, and flagging for review
- Alerts and continuous testing — alerting, feeding failures back into test datasets, and regression testing
flowchart LR
A[Production traffic] --> B[Tracing<br/>capture every interaction]
B --> C[Online evaluations<br/>delayed, possibly sampled]
C --> D[Alerts & reports]
C --> E[Dashboards]
D --> F[New failures & edge cases]
F --> G[Test datasets]
G --> H[Regression testing<br/>on every app change]
Key takeaways
- Four things are trackable in production: product metrics, manual log review, software monitoring, and AI quality monitoring — but product metrics need careful interpretation, e.g. long chatbot conversations may mean users are stuck, not impressed
0:29 - Monitoring tracks metrics defined in advance; observability collects all the data so you can answer questions you didn't anticipate — monitoring is part of observability, not a synonym
1:46 - Production observability is a three-step workflow: tracing, online evaluations, and alerting
2:01 - Unlike traditional tracing, LLM traces matter for their content (user questions, model answers), so product teams need a user-friendly way to inspect them — and masking should protect personal data during manual review
2:42 - Online evaluations run with a delay (e.g. over the last hour of traces) and can be sampled (e.g. 10% of traffic) at high volume, reusing offline methods: LLM judges, semantic similarity, and regular expressions
3:08 - Test datasets remain necessary after launch: every change (new models, bug fixes, prompt adjustments) needs regression testing, and production failures should be added back to the test set to keep it representative
4:16