A really good example of model distillation also comes from DM: their new realtime WaveNet used in Google Assistant. The first WaveNet was ungodly slow due to r

A really good example of model distillation also comes from DM: their new realtime WaveNet used in Google Assistant. The first WaveNet was ungodly slow due to redundant computation; but even after that, it still was not realtime simply because the CNN is too deep and slow. But you need the CNN to be deep & big in order to train good audio generation. Model distillation to the rescue: take a wide fast small CNN and train it to imitate the slow deep WaveNet. Result: WaveNet quality realtime voice generation which can be deployed to the masses.