Evidently AI · 4:05 · https://www.youtube.com/watch?v=ZaOo2dce20A
1. <Untitled Chapter 1>
0:01 [Music] Hi there! Today we're going to talk about the business reasons to do LLM evaluations. It is very clear that creating this evaluation framework is an investment. It costs you some time and money. Let's talk about the business reasons to do this. When you're building an AI product,
2. Investing in AI products
0:16 well, you're already investing in it: in cloud costs, in the infrastructure and in the manpower. And you do this for a reason. Either because you want it to save you money or you want it to earn your money. Like for example, if you have an LLM support chatbot you might expect it to cut your support costs in half. And evaluations is how you make sure that this investment pays off.
0:37 It's just like safety testing for the product you manufacture, or quality assurance for software, or performance reviews for employees. If you have an asset - you must invest in the oversight. Let's talk about four specific reasons why evaluations matter. The first one is quality, the second one is safety. The third one is compliance and the fourth is efficiency. Let's start with quality.
3. Quality
1:00 Quality is not an abstract thing. If your product is good, the people will return and use it and you will earn money. You know, sometimes people get drawn to AI products because of the curiosity. They want to try out the AI feature but they stay only if the product is actually good. And bad products, low quality products can cost you money. Not always just the users.
1:14 Sometimes it's PR disasters or legal risks. You don't want to be the next headline. Next one is safety. In
4. Safety
1:26 some areas errors are not just frustrating, they are dangerous. For example, a medical chatbot can give incorrect advice. This can really affect patient's life. Or in finance the incorrect advice can get you in trouble with regulators. If you're not testing for it - you are gambling with user safety and, ultimately, liability. Another reason is compliance.
1:43 There are already a lot of
5. Compliance
1:49 AI regulations in different parts of the world. And even if your product is not seen as high risk now it might be seen later. Or maybe it will be the enterprise customers that are requiring it, or maybe the competitor setting tone. So you should actually prepare yourself for these regulations, even if you're not yet under them.
2:04 And in this case these regulations often concern on the exact same things we talked about: safety testing, the ability to audit how the system works, production monitoring. So you can prepare ahead by building a solid evaluation system. And finally,
6. Efficiency
2:20 there is efficiency. Without an evaluation system your team will simply slow down. Iterations take longer if you cannot objectively measure if a change improves things. Shipping updates is very stressful if you don't know what they're breaking. You cannot migrate to a new or cheaper model - it's a very real risk.
2:37 There have already been instances when providers discontinued support for models, so you have to switch. But you cannot do this if you don't have proper evaluations in place. Debugging can also be a nightmare. If someone reports an issue, you have no way to trace what caused it or test that you actually fixed it. You don't want to be in that place.
2:55 On the other hand, a good evaluation system is actually a competitive advantage that lets your team move quicker. Because you can try out new ideas faster, you can make changes faster and you can be the first to deliver the features to the market. One side note here. Sometimes people say:
7. Do you need evals for a Proof-of-Concept?
3:11 we don't need evaluations yet because it's just a proof of concept. But here's the thing. Without evaluations, your proof of concept is just a glorified demo. And let's be honest, building demos is already very very easy. So to actually prove something with your concept, you need to test your AI products in albeit limited but realistic conditions.
3:25 You need to make sure that you know how address at least some of the edge cases and you have a system to know how you're measurably improving it. So all in all, even for a proof of concept you do need evaluations. That is actually one of the most important parts of it. To sum up,
8. Summing up
3:46 bottom line is the bottom line. Investing in evaluations is not just a technical decision. It's a business decision. And you want to do this because you want to build features that acquire and retain users, you want to avoid risks, and you want to make sure that you stay compliant and your product actually works in production. So evaluations do matter quite a lot.