Automation does not discount human review — the two should run alongside each other 4:42.
How real companies combine automated and manual evaluation
| Company | Automated | Manual |
|---|---|---|
| Asana | Automated unit tests for AI features 4:42 | Product manager does end-to-end manual review 4:42 |
| Webflow | Automated scores for day-to-day validation 4:42 | Weekly manual scoring to validate all big changes 4:42 |
| GitLab (Duo) | Framework with thousands of prompts and ground-truth answers, tested against daily, plus a smaller version for quick iterations 5:10 | Manual examination of results to find patterns and decide what to focus on next 5:10 |
- GitLab Duo is a suite of AI-powered features validated through this framework
4:42.
Previous: Experiments, stress-testing & regression testing · Next: LLM evals are product-specific