• Core distinction: LLMs are models; LLM applications are what you build using those models — and evaluating each is not the same thing. 0:14
  • The problem benchmarks solve: new models come out all the time, and testing them yourself is neither systematic nor fair. 0:14
  • The community's answer: benchmarks — standardized tests for LLMs, like school exams. 0:31
    • A benchmark has known answers to a set of questions, so model performance can be compared directly. 0:31
    • Beyond simple scoring, there are fancier grading methods and ranking systems, e.g. the Open LLM Leaderboard, which compares different models. 0:31