Alerts and continuous testing in production

Alerts and reports

• Set up alerts on quality thresholds — e.g. hallucination rate rising above your usual baseline 4:01
• Alternatively, periodic reports — e.g. a weekly digest of the most interesting conversations to read 4:01

Test datasets still matter after launch

• New failures and edge cases seen in production should be added to your test datasets 4:01
• Answer to "do I still need test datasets with real production data?" — yes 4:16:
  • you keep changing the app: trying new models, fixing bugs, adjusting prompts 4:16
  • each change needs regression testing — re-run tests to confirm nothing broke 4:32
  • keep making tests more representative by adding production examples 4:32

Why real-time observability, in two reasons

1. Continuous improvement — spot and solve issues efficiently 4:53
2. Peace of mind — confidence that everything is under control and performing smoothly 4:53