9 ms·

Interesting post, but there are several errors, or at least suggestions that don't make sense to me: > Doing so will result in very good evaluation scores and

by glial 10y ago

Interesting post, but there are several errors, or at least suggestions that don't make sense to me: > Doing so will result in very good evaluation scores and make the user happy but instead he/she will be building a useless model with very high overfitting. Testing on your training data doesn't in-and-of-itself lead to overfitting, but it will hide overfitting if it does exist (and is a terrible practice for that reason). > At this stage only models you should go for should be ensemble tree based models. Not sure why this should be the case. Many ensemble models are very memory hungry & slow, Random Forests being a good example. They are flexible and have minimal assumptions, sure, but that doesn't mean you shouldn't try any other modeling technique, especially if you have domain knowledge. > We cannot apply linear models to the above features since they are not normalized. Not true at all. You won't be able to meaningfully compare coefficient magnitudes, but you can certainly apply linear models. > These normalization methods work only on dense features and don’t give very good results if applied on sparse features. Since normalization is done feature-by-feature, there is no reason why this should be true. > generally PCA is used decompose the data If your latent features add linearly, ok, but do they? Is it meaningful to have negative values for your features? If not, consider using something like sparse non-negative matrix factorization. Lastly, using this approach means you have a HUGE model & parameter space to search through. Because of this you will need a ton of data to get meaninigful results. He seems to be treating machine learning methods as a bag of tricks. That's ok so far as it goes, but in my experience it's much more valuable to try and understand your data, and the process that generates it, and then build a model that tries to reflect that data generation process.

danielmorozoff 10y ago
Thank you bringing all these up- whenever something is said to be a one size fits all solution in machine learning, for anything as a matter of fact, I would tend to be wary.
bhntr3 10y ago
> Testing on your training data doesn't in-and-of-itself lead to overfitting, but it will hide overfitting if it does exist (and is a terrible practice for that reason). I read that as pointing out the danger of using training data for hyper-parameter search validation. In practice if you're aggressively tuning your model and feature choices using your training data for validation you're almost certain to have overfitting. I think it actually makes sense to have 3 data sets if you can afford it. > Not sure why this should be the case. Many ensemble models are very memory hungry & slow, Random Forests being a good example. They are flexible and have minimal assumptions, sure, but that doesn't mean you shouldn't try any other modeling technique, especially if you have domain knowledge. I read this as "Try random forests first. If you get good enough results, you're done." He mentions 9 models later and says "In my opinion, and strictly my opinion, the above models will out-perform any others and we don’t need to evaluate any other models." > Not true at all. You won't be able to meaningfully compare coefficient magnitudes, but you can certainly apply linear models. The documentation for StandardScaler says "For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected." > Since normalization is done feature-by-feature, there is no reason why this should be true. StandardScaler re-centers around the mean. That can turn a SparseMatrix into a NotSoSparseMatrix. In a framework that supports only in memory matrices that's probably not what you want, even if it might work. He mentions how to deal with it. Can't comment on PCA. Overall, this seems like practical advice from someone with a lot of experience on Kaggle. Kaggle isn't real life so I'm sure there are better things to do for real models. It's still probably reasonable to take his advice in context instead of assuming it's wrong.
nl 10y ago
A lot of the criticisms here are about inefficiencies in the approach. This is true, but the approach here does generalise well (provided you have sufficient data). It's worth noting that Abhishek is working on the "automatic data science" problem[1] and is viewing thing through that lens. AutoML is an interesting area - there have been workshops at ICML, and good progress is being made in the area. [1] https://arxiv.org/abs/1507.02188 https://arxiv.org/abs/1507.02188
- visarga 10y ago
  This talk is very interesting: they do automatic model selection, pipeline stacking (pick the right sequence of pre-processing stages to place in the pipeline) and hyperparameter optimization. They use meta-learning to improve search speed. Automatic Machine Learning? SciPy 2016 by Andreas Mueller - https://www.youtube.com/watch?v=Wy6EKjJT79M https://www.youtube.com/watch?v=Wy6EKjJT79M
- danielmorozoff 10y ago
  Thanks for posting his paper
psychometry 10y ago
You can overfit the data if you're tuning the hyperparameters outside the CV loop. Ideally, you'd want to have a two-layer CV loop, whether it's k-fold CV, LOOCV, bootstrapping, or whatever. Occasionally this might be computationally infeasible, but it's the most reliable way to do it.
sgt101 10y ago
"..., but in my experience it's much more valuable to try and understand your data, and the process that generates it, and then build a model that tries to reflect that data generation process." There are tasks where I feel happy to accept a black box model; image recognition, speech transcription. In these tasks understanding the model is really important for further research, but it seems to me to be ok to say "look it just works". When it comes to data mining or task like predicting customer responses to offers I believe that arbitrary models that have high predictive power are dangerous. This is because you can't know when changes in your domain (like a new competitor or a shift in demographics) are liable to derail it, and experience shows that this can lead to really bad decisions from something that people have come to trust.

by glial 10y ago

danielmorozoff 10y ago

Thank you bringing all these up- whenever something is said to be a one size fits all solution in machine learning, for anything as a matter of fact, I would tend to be wary.

bhntr3 10y ago

> Testing on your training data doesn't in-and-of-itself lead to overfitting, but it will hide overfitting if it does exist (and is a terrible practice for that reason). I read that as pointing out the danger of using training data for hyper-parameter search validation. In practice if you're aggressively tuning your model and feature choices using your training data for validation you're almost certain to have overfitting. I think it actually makes sense to have 3 data sets if you can afford it. > Not sure why this should be the case. Many ensemble models are very memory hungry & slow, Random Forests being a good example. They are flexible and have minimal assumptions, sure, but that doesn't mean you shouldn't try any other modeling technique, especially if you have domain knowledge. I read this as "Try random forests first. If you get good enough results, you're done." He mentions 9 models later and says "In my opinion, and strictly my opinion, the above models will out-perform any others and we don’t need to evaluate any other models." > Not true at all. You won't be able to meaningfully compare coefficient magnitudes, but you can certainly apply linear models. The documentation for StandardScaler says "For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected." > Since normalization is done feature-by-feature, there is no reason why this should be true. StandardScaler re-centers around the mean. That can turn a SparseMatrix into a NotSoSparseMatrix. In a framework that supports only in memory matrices that's probably not what you want, even if it might work. He mentions how to deal with it. Can't comment on PCA. Overall, this seems like practical advice from someone with a lot of experience on Kaggle. Kaggle isn't real life so I'm sure there are better things to do for real models. It's still probably reasonable to take his advice in context instead of assuming it's wrong.

nl 10y ago

A lot of the criticisms here are about inefficiencies in the approach. This is true, but the approach here does generalise well (provided you have sufficient data). It's worth noting that Abhishek is working on the "automatic data science" problem[1] and is viewing thing through that lens. AutoML is an interesting area - there have been workshops at ICML, and good progress is being made in the area. [1] https://arxiv.org/abs/1507.02188 https://arxiv.org/abs/1507.02188

visarga 10y ago
This talk is very interesting: they do automatic model selection, pipeline stacking (pick the right sequence of pre-processing stages to place in the pipeline) and hyperparameter optimization. They use meta-learning to improve search speed. Automatic Machine Learning? SciPy 2016 by Andreas Mueller - https://www.youtube.com/watch?v=Wy6EKjJT79M https://www.youtube.com/watch?v=Wy6EKjJT79M
danielmorozoff 10y ago
Thanks for posting his paper

psychometry 10y ago

You can overfit the data if you're tuning the hyperparameters outside the CV loop. Ideally, you'd want to have a two-layer CV loop, whether it's k-fold CV, LOOCV, bootstrapping, or whatever. Occasionally this might be computationally infeasible, but it's the most reliable way to do it.

sgt101 10y ago

"..., but in my experience it's much more valuable to try and understand your data, and the process that generates it, and then build a model that tries to reflect that data generation process." There are tasks where I feel happy to accept a black box model; image recognition, speech transcription. In these tasks understanding the model is really important for further research, but it seems to me to be ok to say "look it just works". When it comes to data mining or task like predicting customer responses to offers I believe that arbitrary models that have high predictive power are dangerous. This is because you can't know when changes in your domain (like a new competitor or a shift in demographics) are liable to derail it, and experience shows that this can lead to really bad decisions from something that people have come to trust.