3 ms·
author here; yea the goal is to try and get the student to surpass the teacher, but if it can't, this is the best way to get close. For these contrastive losses
by corbyrosset 3y ago
author here; yea the goal is to try and get the student to surpass the teacher, but if it can't, this is the best way to get close. For these contrastive losses, our intuition is that the model isn't trying to emulate the teacher so much as learning from the 'delta' between itself and the teacher