If you do it during training you automatically check that the compressed model still do what you want after compression, and can even correct some small issues

If you do it during training you automatically check that the compressed model still do what you want after compression, and can even correct some small issues by continuing the training on the smaller model. Edit: In general the more a compression function understands of what your goals are the better, so it is naturally advantageous to make the compression function look like training since then it is fully aware of what to optimize for.