When distilling models for speed, you get a better win from removing decoder parameters, since they are run in serial, than encoder parameters. For example see

When distilling models for speed, you get a better win from removing decoder parameters, since they are run in serial, than encoder parameters. For example see this work https://arxiv.org/abs/2006.10369 https://arxiv.org/abs/2006.10369 - paper co-author