3 ms·

I'm curious on this claim. Feels like any model can over fit.

• HelloNurse 4y ago
  With algorithms that, for each additional input entry, either add a predictable amount of complexity to improve fitness (e.g. another tree split) or decide that the model is already fit enough, one needs to do something deliberate or very stupid to overfit data compare to neural networks where overfitting is hard to detect, let alone prevent quantitatively, and that are sometimes designed to overfit in order to interpolate "intelligently" between samples.
• marcosdumay 4y ago
  A model can only overfit if it has enough parameters. But indeed, any model with enough parameters can overfit. The thing about deep learning is that the number of parameters is astronomical.
  • disgruntledphd2 4y ago
    Hence, the interesting question is why deep learning models don't overfit all the time.
    • DougBTX 4y ago
      This is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
      • disgruntledphd2 4y ago
        Agreed, but it doesn't really explain it in any real sense.
        • mr_toad 4y ago
          “However, in the over-parameterized regime, there are many models that fit the train set and there exist such good models.” One explanation is that a very large model effectively fits several models and penalises over-fit models itself.
        • skinner_ 4y ago
          Work is ongoing, this is a whole field of research now, a good keyword to search for is "implicit regularization".
      • marcosdumay 4y ago
        I really like how AI research is all about empirical analysis of mathematics.
    • gnramires 4y ago
      The answer is really very simple: because we use training and test sets, and generally use iterative training methods. As soon as test set error plateaus or starts to degrade, you stop training. This is called early stopping and can be considered a form of regularization. In theory it would seem a neural network (with early stopping "correctly" implemented) can never overfit, although I suppose in practice it can vary a little (because of test variance, too large step sizes, etc.). It's possible to train huge overparametrized networks this way. One big problem I can see with this approach (maybe compared to over forms of regularization) is that it might be very space/memory inefficient. But if you've got a big budget I think it's a surefire way to make sure your model performs as well as possible (without having to tune hyperparameters too much or deal with local minima that probably happens at near-capacity networks).
• bbstats 4y ago
  Random Forests can overfit, but with sufficient data (and no data leakage) it is difficult to do so. The final output of a random forest regression for example is just the average of all the final prediction nodes of the individual trees, which thanks to bootstrapping are decorrelated. So - adding more trees does not tend to alter predictions very much.