They don't justify it explicitly, but they do talk about using the distilled model as an assistant for the original. With the encoder precisely the same for bo

They don't justify it explicitly, but they do talk about using the distilled model as an assistant for the original. With the encoder precisely the same for both you only need to additionally load the distilled decoder layers for a 2x speedup with the same accuracy as the original.