3 ms·

> One way or another, overfitting is no joke and it's definitely no urban legend. Can you give an example of where overfitting happened and was successfully co

> One way or another, overfitting is no joke and it's definitely no urban legend. Can you give an example of where overfitting happened and was successfully corrected for?

red75prime 9y ago
That's why testing and validation sets exists. Overfitting is prevented by ending learning process when error on validation set starts growing. It is a standard procedure, so it's unlikely there's a recent case of overfitting slipping into production. If irrelevant feature is present in all class samples, then it is not a fault of NN to use it as a class feature, it's bad data.
- aaron695 9y ago
  So you would agree overfitting is not a real issue and talking about it is distracting from NN? My question is are people using overfitting as an excuse of a what is instead a badly made NN. If you are smart enough to create a NN that can tell if it's sunny or not then tanks would also be possible. But if your NN just sucks than blaming overfitting is a convenient out.
  - YeGoblynQueenne 9y ago
    >> My question is are people using overfitting as an excuse of a what is instead a badly made NN. Overfitting is a major issue in machine learning and it's an inherent characteristic of learning from examples and not the result of a mistake, or of poor practice. There are special techniques developed explicitly to reduce overfitting- early stopping (what red75prime above, describes), regularisation, bagging (in decision trees) etc. A lot of work also goes into ensuring measures of learning performance don't mistake overfitting for successful learning (e.g. k-fold cross validation). I'm sorry that I don't have time to track down a good source for a discussion of the bias-variance tradeoff and overfitting. You can start at the wikipedia page [https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff] and follow the links. In short- a model that learns to reproduce its example data with very high fidelity, risks generalising poorly, whereas a model that generalises well may have high training error. Linear classifiers in particular are high-bias, whereas nonlinear learners, like multi-layered neural networks or decision trees, are high-variance. The problem is real, it's a big bugbear and you won't find any specialist who dismisses it, or who considers it "not a real issue".
  - jamez1 9y ago
    Building a robust process that creates a model fitting signal and not noise is very tough. It becomes philosophical for some problems, such as forecasting. Over-fitting is dismissed as an amateur mistake when it really is an endemic problem you are constantly battling no matter how good you are.

> One way or another, overfitting is no joke and it's definitely no urban legend. Can you give an example of where overfitting happened and was successfully corrected for?

red75prime 9y ago

That's why testing and validation sets exists. Overfitting is prevented by ending learning process when error on validation set starts growing. It is a standard procedure, so it's unlikely there's a recent case of overfitting slipping into production. If irrelevant feature is present in all class samples, then it is not a fault of NN to use it as a class feature, it's bad data.

aaron695 9y ago
So you would agree overfitting is not a real issue and talking about it is distracting from NN? My question is are people using overfitting as an excuse of a what is instead a badly made NN. If you are smart enough to create a NN that can tell if it's sunny or not then tanks would also be possible. But if your NN just sucks than blaming overfitting is a convenient out.
- YeGoblynQueenne 9y ago
  >> My question is are people using overfitting as an excuse of a what is instead a badly made NN. Overfitting is a major issue in machine learning and it's an inherent characteristic of learning from examples and not the result of a mistake, or of poor practice. There are special techniques developed explicitly to reduce overfitting- early stopping (what red75prime above, describes), regularisation, bagging (in decision trees) etc. A lot of work also goes into ensuring measures of learning performance don't mistake overfitting for successful learning (e.g. k-fold cross validation). I'm sorry that I don't have time to track down a good source for a discussion of the bias-variance tradeoff and overfitting. You can start at the wikipedia page [https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff] and follow the links. In short- a model that learns to reproduce its example data with very high fidelity, risks generalising poorly, whereas a model that generalises well may have high training error. Linear classifiers in particular are high-bias, whereas nonlinear learners, like multi-layered neural networks or decision trees, are high-variance. The problem is real, it's a big bugbear and you won't find any specialist who dismisses it, or who considers it "not a real issue".
- jamez1 9y ago
  Building a robust process that creates a model fitting signal and not noise is very tough. It becomes philosophical for some problems, such as forecasting. Over-fitting is dismissed as an amateur mistake when it really is an endemic problem you are constantly battling no matter how good you are.