4 ms·
Frankly, I disagree. I think the conclusions are domain-specific. In my field (machine learning/informatics in a medical context), it is very often the case th
by et2o 9y ago
Frankly, I disagree. I think the conclusions are domain-specific.
In my field (machine learning/informatics in a medical context), it is very often the case that simple models are better as they generalize to the next set of patients better, even if a more complex model may fit the test dataset better. In some contexts, it requires a lot of hubris to think that your model is really describing reality and free from the intricacies of your particular dataset.
To generalize, complex models tend to overfit on the small datasets we typically have. One example of this is the difficulty in transferring transcriptomic models for risk prediction or other biological characteristics between clinical trials; a model with a LOOCV AUC of 0.9 can get used on a different clinical trial and perform quite poorly. Deep learning certainly has its place, but there is nothing inherently superior about a more complex model.
- RandyRanderson 9y agoOf all the top Kaggle competition solutions I've looked at, very few could not be done on a laptop. I can think of only one exception and he was using one computer with 100GB+ RAM. Doubling your data will likely only get you maybe 10% more accurate than the last increment. This seems like a lot until you realize that you went from a RMSE of 0.87567 to 0.87566. This doesn't make any material difference to your bottom line other than it takes more compute resources and more complexity in terms of coding. In this sense it may be actually costing you more to maintain a larger model and also the results may be less clear. I would encourage ML practitioners to try the following exercise: half a problem's data and re-train your models. I think you'll find that your training will typically be much faster and the difference in the performance will be minimal.
- rubatuga 9y agoCould not agree with your statements more. Doing statistical modelling with a small subset or too many independent variables leads to problems, which seems to be similar to what you are describing. Although I know nothing about machine learning, it would make sense that would be the case. This is off topic, but I'm an undergrad in bioinformatics, and was wondering if you have any tips into where new research may head? I will have to partner with a PI later this fall, and was wondering what kind of project I would pursue