4 ms·

In my experience, overfitting also helps in case of small data problems. After all, there is a reason that you do only have few training data samples and the ch

In my experience, overfitting also helps in case of small data problems. After all, there is a reason that you do only have few training data samples and the chance that any new prediction sample is close to the existing training ones is higher. In other words, for small data its more about specialization than generalization. Of course this has to be evaluated case specific.

nri 4y ago
Interesting, that advice is exactly the opposite of the common wisdom. You mean overfitting as in aggressively maximizing your crossvalidation scores? How do you decide for which problems that is a good approach and for which a more conservative approach is better?
- smartmic 4y ago
  Well, I mean not actively maximizing your CV score, just accepting normally insufficient CV scores. For example, in a tree based regression with a very small number of training samples, the leaves will almost resemble some of the training samples. A good generalization might not be possible at all. But this is not too bad if you know that your prediction samples have high similarity with at least one of the training samples. In the end, this is an edge case for Machine Learning but goes more into the direction of Expert Systems.
  - isoprophlex 4y ago
    Is this a complicated way of saying "just use k-nearest neighbors"?
    - MrMan 4y ago
      I think its implying that ML often works because the manifold assumption is partially correct. so samples will lurk in the same neighborhood.
    - crabbygrabby 4y ago
      In a lot of cases knn doesn't model systems appropriately unless you understand the space very well. At least knn offers some sort of theoretical assumption of the bayes error rate though so it's a decent tool.
workingon 4y ago
I would take the opposite approach here. Instead of overfitting, bake probability into your model. Whether using a Bayesian weights or using an ad-hoc version (I.e. dropout and batch normalization), you can do your predictions in ensemble and look at the deviation of predictions. the combination of this method and a small dataset usually leads to wider ranges of prediction, which can be interpreted as a model uncertainty. When using these techniques we have found smaller datasets lead to more uncertainty, but it often will bound the error. I think this is much more useful than assuming a small dataset means an overfit model will do good on future data.

nri 4y ago

Interesting, that advice is exactly the opposite of the common wisdom. You mean overfitting as in aggressively maximizing your crossvalidation scores? How do you decide for which problems that is a good approach and for which a more conservative approach is better?

smartmic 4y ago
Well, I mean not actively maximizing your CV score, just accepting normally insufficient CV scores. For example, in a tree based regression with a very small number of training samples, the leaves will almost resemble some of the training samples. A good generalization might not be possible at all. But this is not too bad if you know that your prediction samples have high similarity with at least one of the training samples. In the end, this is an edge case for Machine Learning but goes more into the direction of Expert Systems.
- isoprophlex 4y ago
  Is this a complicated way of saying "just use k-nearest neighbors"?
  - MrMan 4y ago
    I think its implying that ML often works because the manifold assumption is partially correct. so samples will lurk in the same neighborhood.
  - crabbygrabby 4y ago
    In a lot of cases knn doesn't model systems appropriately unless you understand the space very well. At least knn offers some sort of theoretical assumption of the bayes error rate though so it's a decent tool.

workingon 4y ago

I would take the opposite approach here. Instead of overfitting, bake probability into your model. Whether using a Bayesian weights or using an ad-hoc version (I.e. dropout and batch normalization), you can do your predictions in ensemble and look at the deviation of predictions. the combination of this method and a small dataset usually leads to wider ranges of prediction, which can be interpreted as a model uncertainty. When using these techniques we have found smaller datasets lead to more uncertainty, but it often will bound the error. I think this is much more useful than assuming a small dataset means an overfit model will do good on future data.