Costs of NOT having evaluations
- The team simply slows down: iterations take longer when you cannot objectively measure whether a change improves things.
2:20 - Shipping updates becomes stressful because you don't know what they're breaking.
2:20 - You cannot migrate to a new or cheaper model — called out as "a very real risk".
2:20 - Providers have already discontinued support for models, forcing switches — impossible to do safely without proper evaluations in place.
2:37 - Debugging becomes a nightmare: when someone reports an issue you have no way to trace what caused it, nor to test that you actually fixed it.
2:37
The flip side
- A good evaluation system is a competitive advantage: the team moves quicker, tries new ideas faster, makes changes faster, and can be first to deliver features to the market.
2:55