Evidently AI · 1:31 · https://www.youtube.com/watch?v=rHs0sP7b5fM
0:01 [Music] Hi and welcome to the course on LLM evaluations. I'm Elena and I will be your instructor for the course. If you're working on an LLM-powered product or just thinking about it - you're in the right place. This course is meant as a high-level introduction to the topic. You don't need to be an expert and there will be no coding.
0:15 All you need is a general idea of what it takes to build a product with LLM inside and some curiosity to learn more. Here's what we're going to talk about in the course. First, we'll explain what an LLM is and what's an LLM-powered product.
0:30 Then, we're going to talk about LLM benchmarks that you can use to evaluate large language models. After that, we'll start talking about products: we'll discuss how to build an evaluation dataset, how to write your test cases and which methods and metrics you can use for your evaluations.
0:45 Specifically, we're going to talk about LLM as a judge which is a fascinating but slightly controversial topic. We'll also talk about LLM safety and things like hallucinations and jailbreaks and how you can protect against them. We'll also touch upon more complex topics like evaluating AI agents. After that, we're going to talk about what happens after you launch: production observability, monitoring and guardrails.
1:09 And finally we'll talk about the business reasons to invest in AI evaluations. I'll try to keep each video short and link to the extra materials that you can study on your own. There will also be a bonus video where we're going to see a practical example of how to run LLM evaluations with no code. Let's start! See you you in the next videos.