LLM evals are product-specific

Every evaluation framework is unique to its product

Each framework is very product-specific: know your user, understand the risks you're protecting against, and have a clear idea of what "good" means for your use case 5:25.
Evaluations should therefore be designed together with domain experts 5:25.
Evaluations are only partially a technical problem (how to set them up and run them through the app); the contents of the evaluations matter just as much 5:25.
Conclusion: evals should be done by the complete team, not just engineers 5:43.

How to design the datasets, and which exact methods to use to validate and evaluate them 5:43.