Three reasons the community keeps producing new benchmarks:

  1. Many skills to evaluate — there are a lot of distinct capabilities worth testing. 1:51
  2. Benchmarks become outdated — as new models are released, harder, more complex benchmarks are needed. 1:51
  3. Training contamination — models can inadvertently memorize benchmark data during training, so fresh benchmarks are always required. 1:51

Caveat that leads into the next topic: benchmarks don't necessarily test what you care about when building an LLM-powered product. 2:07