4 ms·

Still only 33% though. Which is impressive for a 7B model, but the student has not yet surpassed the teacher

• danielcampos93 3y ago
  If you only emulate the teacher can you ever become an master?
  • tosh 3y ago
    It works in many competitive fields
    • Twirrim 3y ago
      Surely it needs more than just emulating the teacher, though, it would require exploring beyond those limitations?
      • CapeTheory 3y ago
        But eventually your human mentor diminishes with age. And then you can crush them.
• corbyrosset 3y ago
  author here; yea the goal is to try and get the student to surpass the teacher, but if it can't, this is the best way to get close. For these contrastive losses, our intuition is that the model isn't trying to emulate the teacher so much as learning from the 'delta' between itself and the teacher