Not for DL models, but I was exploring doing this in a specific setting (I've posted about this earlier). For small-sized models (for some reasonable definition

Not for DL models, but I was exploring doing this in a specific setting (I've posted about this earlier). For small-sized models (for some reasonable definition of size, e.g., depth of a decision tree, or # trees in a gradient boosting forest, or # non-zero coefficients in a linear model), I realized that you could make them even smaller while retaining their accuracy by selectively presenting training data to them, i.e., ignore some training data points, repeat certain others. See [1] - the x-axis shows the original model size and the y-axis shows the model size obtained by this process at the same or better accuracy. Interestingly, this meant that the conventional wisdom that the test and train distributions have to be identical for optimal held-out performance, is not true at small model sizes. It is true as models grow larger - and I was explicitly able to show this. See [2] - where the x-axis is model size, and the y-axis measures (on a scale of 0-1) how close is the optimal training distribution to the test distribution. The different lines are for different datasets. These images are from the paper here [3]. I have a library too [4]; that is in need of updates - it works today as-is though, but please use the latest minor release if this of interest [5]! [1] https://imgur.com/a/NheK49Z https://imgur.com/a/NheK49Z [2] https://imgur.com/a/N53GeNI https://imgur.com/a/N53GeNI [3] https://arxiv.org/pdf/1906.06852.pdf https://arxiv.org/pdf/1906.06852.pdf [4] https://compactem.readthedocs.io https://compactem.readthedocs.io [5] As of writing the install command should be `pip install compactem==0.9.9rc2`