Train for the environment

  1. Pre-training builds general knowledge from internet text and other data. 34:29
  2. Supervised fine-tuning teaches assistant or agent behavior. Examples of the actual harness help the model learn its tools and environment. 34:29 34:58
  3. Reinforcement learning can improve instruction following, skill selection among hundreds of skills, and browser interaction. 34:58 35:25

Serve different roles differently

  • Optimize the main conversational agent for low latency so it responds promptly to the user. 35:25
  • Long-running helpers tolerate different tradeoffs. For a background task, Lee uses 10 versus 12 minutes as an example of a difference that may matter little to the user. 35:50

Two improvement loops

Daily product-fix loop connected to a slower model-training loop

Figure 9. Failures become evaluations for daily product fixes and later model training. New model capabilities introduce new behaviors to evaluate. Source: leerob, 36:38.

  • The inner loop runs frequently: capture a failure, make it an evaluation, fix the harness, prompt, code, or inference behavior, verify the fix against evaluations, and ship. An incorrect tool call is the example failure. 36:13 36:38
  • The outer loop trains new models over longer cycles, using failures and evaluations to improve the behaviors the product needs before substituting the model into the product. 36:38
  • New capabilities expose new interaction problems. Lee identifies group chats and bot-to-bot communication as emerging use cases with limited training data that need explicit behavioral training. 37:02 37:25