Video cover

Evidently AI · 3:07 · Watch on YouTube

Short educational explainer that sets the ground for an LLM evaluation course: before evaluating anything, it defines what is being evaluated. It covers what an LLM is and how you interact with it, what makes software an "LLM-powered product" (from single-prompt features to RAG systems and agents), and closes with real-world examples of companies shipping LLM features — which is where the need for evaluation begins.

Topics

  1. Introduction
  2. What's an LLM?
  3. What's an LLM-powered product?
  4. Real-world examples of LLM-powered products

Overview

flowchart LR
    A[LLM<br/>prompt in, text out] --> B[LLM-powered product<br/>software using an LLM inside]
    B --> C1[Simple: single prompt]
    B --> C2[Prompt chains]
    B --> C3[RAG: external data]
    B --> C4[Agents: plan, tools, memory]
    B --> D[Is it working?<br/>→ Evaluation / evals]

Key takeaways

  • An LLM is an AI model trained on huge amounts of text; unlike previous ML waves, you typically don't train it yourself — you take an available model (maybe fine-tune it) and use it as is 0:15.
  • LLMs handle both generative tasks (open-ended output: summaries, code, emails, translation) and simpler predictive tasks (classifying text by sentiment or topic) 0:35.
  • An LLM-powered product is any software that uses an LLM inside its functionality — either LLM features added to an existing product or entirely new LLM-based applications (chatbots, assistants, coding tools) 1:06.
  • Complexity scales: from a single well-crafted prompt, to prompt chains, to RAG (giving the LLM external data, which adds a database and search logic), to agents that plan steps, use tools, and keep memory 1:30 1:48.
  • Real companies already ship these: StitchFix/NextDoor (marketing text), Salesforce (Slack summaries), Vimeo (video summaries), Google and Incident.io (incident summaries), Dropbox (Q&A over files), DoorDash (support chatbot) 2:21.
  • Once you build such a product, you inevitably ask "is it working as expected, and how do I improve it?" — that is where evaluation ("evals") starts, the subject of the next videos 2:42.