Video cover

Evidently AI · 5:59 · Watch on YouTube

Summary

A big-picture introduction to evaluating LLM-powered applications, focused on the pre-deployment phase. It explains why LLM testing differs from traditional software testing (non-determinism), frames quality as capabilities plus risks, and walks through the maturity path from informal vibe checks to manual labeling to a fully automated evaluation system built on test datasets and scoring methods. It closes with how that system enables experiments, red-teaming, and regression testing, illustrated with real practices from Asana, Webflow, and GitLab.

Topics

  1. Introduction & when we need LLM evals
  2. Why evaluating LLMs is hard
  3. What is LLM product quality
  4. From vibe checks to manual labeling
  5. Automated evaluations
  6. Experiments, stress-testing & regression testing
  7. Examples from Asana, Webflow, GitLab
  8. LLM evals are product-specific

The evaluation maturity path

flowchart TD
    A[Vibe checks<br/>informal, not systematic] --> B[Manual labeling<br/>systematic but doesn't scale]
    B --> C[Automated evaluations<br/>test dataset + scoring methods]
    C --> D[Experimental comparisons<br/>prompts, parameters, models]
    C --> E[Stress-testing / red-teaming<br/>risky and unexpected inputs]
    C --> F[Regression testing<br/>changes don't break other scenarios]
    D & E & F --> G[Product-specific eval framework<br/>automated + human review together]

Key takeaways

  • LLM apps are non-deterministic — the same input can yield different outputs — so testing means checking that the range of possible outputs stays within expectations, not that the system "works" 0:39 1:00.
  • LLM product quality reduces to two axes: capabilities (accurate, useful, helpful responses) and risks (safe, non-harmful outputs) 1:16.
  • Vibe checks aren't systematic and manual labeling doesn't scale; the destination is your own evaluation system — a test dataset with ground truth plus scoring via exact match, semantic similarity, or open-ended criteria (including LLM-as-judge for subjective qualities like tone) 1:40 2:14 2:30 3:08.
  • LLMs are brittle: small prompt tweaks can have unexpected side effects, so regression-test after every change — even by replaying last week's production data under the new prompt 4:21.
  • Automation never replaces human review: Asana pairs automated unit tests with PM end-to-end review, Webflow pairs daily automated scores with weekly manual scoring, and GitLab tests Duo daily against thousands of prompts with ground-truth answers, then manually examines results for patterns 4:42 5:10.
  • Eval frameworks are product-specific — defining "good" requires domain experts and the complete team, because the contents of the evaluations matter as much as the technical setup 5:25.