Evidently AI · 3:07 · https://www.youtube.com/watch?v=B7fl6He1lxM

1. <Untitled Chapter 1>

0:02 [Music] Hi! Before we talk about LLM evaluations, let's clarify what we are evaluating. In this video we're going to discuss what's an LLM and what's an LLM-powered product. An LLM is a large

2. What’s an LLM?

0:15 language model. It's an AI model trained on huge amounts of text data. Unlike previous waves of machine learning you don't usually train these models yourself. You just take an available model, maybe fine-tune it but generally use it as is for your purposes. There are commercial available models like those from Open AI, Anthropic or Google and openly available models like from Mistral and Llama. To interact with the model you write a prompt.

0:35 It's a set of text instructions. The model then outputs text in return. You can ask for a lot of things like extract information, write summaries, generate code, write emails, translate things and so on. Many of these tasks are generative: the output is open-ended like a written text. But LLMs can also handle simpler predictive tasks like classifying text by sentiment or topics.

1:00 Now,

3. What’s an LLM-powered product?

1:06 that's an LLM. What's an LLM powered product? Well, this is any software that uses LLM inside its functionality. You can add LLM-powered features to existing products. For example, give your users the option to talk to their data. Or, you can create completely new applications like chatbots, assistants, coding tools and so on that are powered by LLMs entirely.

1:30 Some of these features are quite simple. Say, to generate e-commerce item descriptions you might need a single well-crafted prompt. But often things can get more complex. For example, you might need to chain multiple prompts together. Or you might want to use retrieval augmented generation also known as RAG. This means giving an LLM access to external data.

1:48 Think of a support chatbot that can pull help centre articles to give the response. This adds components like a database and search logic to your app. You can also create AI agents that solve multi-step tasks. They can plan the steps, use tools like internet search and keep memory. Think of an AI research assistant that can write a report for you. Inside these are complex graphs and chains of prompts.

2:14 Here

4. Real-world examples of LLM-powered products.

2:21 are a few real world examples of how companies are using Large Language Models to create products. StitchFix and NextDoor create marketing texts with LLMs. Salesforce summarizes Slack conversations. Vimeo summarizes videos. Google and Incident.io summarize security incidents. Dropbox lets you ask questions about files. DoorDash built an LLM-based support chatbot.

2:42 I'll link to more examples like this below. If you're building a similar LLM-powered product or feature, at some point you will wonder: Is it working as expected? What can I do to improve it? That's when you start caring about evaluation or evals, how community calls it. Let's talk about it in the next videos.