4 ms·
Overfitting is a quantifiable problem. If you're not doing robust data segregation and CV you're not even engaging in elementary ML practices.
by nkozyra 8y ago
Overfitting is a quantifiable problem. If you're not doing robust data segregation and CV you're not even engaging in elementary ML practices.
- juliend2 8y agoWhat do you mean by CV? I'm not familiar with those terms. Thank you.
- lostcolony 8y agoNever seen the acronym (not really in the space) but I assume cross validation.
- nkozyra 8y agoAs sibling points out, cross validation, which is the front-line approach to avoiding overfitting for supervised classification problems.
- DEADBEEFC0FFEE 8y agoI assumed Code Versioning so that if you have robust data segmentation you have less uncertainty about the impact of change. However, I'm a tourist here and hope OP comes back to share.
- BenFrantzDale 8y agoI assumed Computer Vision.
- tomrod 8y agoCross-validation: testing model fit on non-training data
- sbhn 8y agoCamouflaged Vacuity
- thousandautumns 8y agoIt means cross validation. It essentially means is a way of simulating how well your model will do when it encounters real world data. When building a model, you divide your data into two parts, the training set and the testing set. The training set is usually larger (~80% of your original data set, although this can vary), and is used to fit your model. Then, you use the remaining data you set aside for the testing set by using your model to generate predictions for that data, and comparing it to the actual values for that data. You can then compare the accuracy of the model for the training and testing sets to get an idea if your model generalizes well to the real world. If, for example, you find that your model has an accuracy of 95% on the training data, but 60% on your testing data, that means your model is overly tuned into features of the data used to build the model that may not actually be helpful for prediction in the real world.
- janekm 8y agoOnly if the training data you got is representative of all future use cases. Good luck with that.
- nkozyra 8y agoThat's not hard and fast, though. While no model is perfect, robust models can "handle" outliers. Worst case, you know when it happens and train with more a priori.
- nerdponx 8y agoWorse case? More like best case. It's not about outliers. Let's say you're at a startup and you fit some model to your first 30 customers. It works great for your next 10 customers, but fails dramatically for your first enterprise client. Why? Because the enterprise client was fundamentally different from your previous 40 customers. If you fit your model on a population in which the relationship looks one way, then try to apply your model to a population with a different relationship, it will fail. Machine learning and statistics are both application of the same principles of probability and information theory. They work (for the most part) by modeling the world capturing the relationships between random variables. A random variable can be any natural process that we can't express in precise terms, so we express it in probabilistic terms. This is the same principle underlying the premise that "past results do not guarantee future success." The relationships between random variables in the world that affect success in anything -- stock market performance, legal outcomes, etc. -- might not be the same tomorrow as they are today. And that's not even a matter of overfitting. That's just your ever-present real-world threat of having all your modeling work invalidated by forces outside your control. Overfitting happens when you, the data scientist, fit your model to random noise in the training data. An overfitted model will have bad generalization performance on held-out samples, even from the same population. It's not always easy or possible to detect overfitting, especially with small training sets.
- tomrod 8y agoClose. Extrapolation is possible using structural theories rather than only reduced form models.