This paper seems to me like it's trying to overstate its usefulness. I don't see the 4x reduction in training costs. And it also seems to require that the teach

This paper seems to me like it's trying to overstate its usefulness. I don't see the 4x reduction in training costs. And it also seems to require that the teacher and the student models have the same number of decoder layers, which for all common transformer models just isn't the case. It also doesn't work to train a transducer (a common real-time deployment architecture) based on a transformer teacher. In short, I can't imagine ever having a training situation where this paper would be applicable.