Check out this work from Rich Caruana & collaborators on model compression: http://www.cs.cornell.edu/~caruana/compression.kdd06.pdf http://www.cs.cornell.edu/~

Check out this work from Rich Caruana & collaborators on model compression: http://www.cs.cornell.edu/~caruana/compression.kdd06.pdf http://www.cs.cornell.edu/~caruana/compression.kdd06.pdf which was a precursor to the model distallation work from Geoff Hinton: https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531