The evaluation process typically matures in stages as the product evolves.
Stage 1 — Vibe checks
- The first application version usually gets evaluated informally: try it out, look at the outputs, decide if you like them — what the community calls vibe checks
1:40. - Limitation: not systematic
1:40.
Stage 2 — Manual labeling / annotation
- A more formal review process: you or someone on your team grades responses and systematically decides if they're good or bad
1:59. - Best practice: add annotations explaining each decision — they later help you design better metrics
1:59. - Limitation: doesn't scale — you can't grade every response every time, so you need an automated way of labeling responses
2:14.
Previous: What is LLM product quality · Next: Automated evaluations