3 ms·

Distilling ModernBERT into a static model doesn't work

• stephantul 2y ago
  Hello everyone, here's some experiments we tried in distilling a ModernBERT model into a smaller model. Hope you enjoy!
  • yorwba 2y ago
    Are most of the words in WordSim split into multiple BPE tokens? In that case I'd be curious if undistilled ModernBERT fares better on the task when it embeds all tokens in a word simultaneously. And whether results differ depending on how many tokens a word has. Because alternatively, it could also be that the "no context" case is simply out of distribution for ModernBERT and it really needs some longer text to work with.
    • stephantul 2y ago
      That's a good point! In many WordSim tasks, the words are pretty rare on purpose, so this is very likely. But it provides a good way of testing this hypothesis. Nice, we'll check that out.