What the video covers
- Big-picture introduction to evaluating LLM applications: what quality means for LLM-powered apps, the general process for running evaluations, and specific scenarios like regression testing and prompt comparison
0:02.
The two moments when evaluations matter
- Pre-deployment — while building the product, to make sure it's good enough before shipping
0:22.
- In production — once deployed, to make sure it works as expected in the real world
0:22.
- This video focuses on pre-deployment evaluation
0:22.
Next: Why evaluating LLMs is hard