• It is not a metric in the sense that precision or recall is — it's an evaluation method that approximates your human judgment 2:09
  • Mental model: it's like an intern you ask to grade responses following certain instructions — and you must actually give those instructions 2:09
  • With a human intern you'd show examples: "this is a good response", "this is a bad response — and here's why"; an LLM judge works exactly the same way 2:26
  • Consequence: the quality of the judge lives in the evaluation prompt — you must write a very good one to make the labelling work for your case 2:26