Non-determinism is the root difficulty
- LLMs and LLM-powered apps are non-deterministic: the exact same input can produce different outputs
0:39. - This is a feature (creativity, generating new things) and a bug (it generates problems) at the same time
0:39. - Example: a customer support chatbot should not invent answers — its responses must be grounded
0:39.
What this means for testing
- LLM testing is not about checking that the system "works" (as in traditional software)
1:00. - It is about checking that the range of possible outputs stays within your expectations
1:00.
Previous: Introduction & when we need LLM evals · Next: What is LLM product quality