3 ms·

Have not read the paper yet but why do they only cut the decoder and not the encoder?

• regularfry 3y ago
  They don't justify it explicitly, but they do talk about using the distilled model as an assistant for the original. With the encoder precisely the same for both you only need to additionally load the distilled decoder layers for a 2x speedup with the same accuracy as the original.
• srush 3y ago
  When distilling models for speed, you get a better win from removing decoder parameters, since they are run in serial, than encoder parameters. For example see this work https://arxiv.org/abs/2006.10369 https://arxiv.org/abs/2006.10369 - paper co-author