5 ms·

This is a great post as far as it goes. While Abhishek mentions Keras for neural networks (and Keras is a great Python library), he doesn't really go into the c

by vonnik 10y ago

This is a great post as far as it goes. While Abhishek mentions Keras for neural networks (and Keras is a great Python library), he doesn't really go into the cases where deep neural networks are more useful than other algorithms, and how that changes a data scientist's workflow. DNNs are really well suited to unstructured data, which isn't the kind he highlights. One reason for that is because they automatically extract features from data using optimization algorithms like stochastic gradient descent with backpropagation. What that means is they bypass the arduous process of feature engineering. They help you get around that chokepoint, so that you can deal with unstructured blobs of pixels or blobs of text. Because unstructured data is most of the data in the world, and because DNNs excel at modeling it, they have proven to be some of the most useful and accurate algorithms we have for many problems. Here's an overview of DNNs that goes into a bit more depth: http://deeplearning4j.org/neuralnet-overview.html http://deeplearning4j.org/neuralnet-overview.html

Eridrus 10y ago
I think the whole "you don't need to engineer features" thing can be a bit oversold, in that you don't need to build explicit feature extractors, but you do still need to design your architecture to fit your problem domain. It's more general than traditional feature extraction, but if you keep working on a single problem for long enough you may find that while deep learning can be a great starting point, you eventually want to add more hand-coded features if you really have some insight that the neural net isn't fully capturing. I'm personally quite interested to see if we find meaningful ways of constructing differentiable feature extractors that we can train conjointly with a neural net.
bhntr3 10y ago
I think the post represents the typical data scientist's approach to a machine learning problem pretty well. Deep learning is exciting but, for practical problems, most people aren't using it yet. XGBoost is still dominant in Kaggle competitions. Lots of companies with lots of data still use simple single machine models. So, I think it's a little misleading to say that deep learning eliminates the need for feature engineering. If we're going to provide business value to a company, we're often dealing with somewhat structured data and not "unstructured blobs of pixels or blobs of text". There are several simple linear models with highly engineered features in production where I work that resist attempts to replace them with more clever models and less feature engineering. Deep learning is awesome and there may be a time when it solves every problem. But let's not oversell it. For now, if someone wants to do machine learning and get paid for it, they're more likely to see their colleagues using the techniques in this article than training deep multi-layer neural nets.
- nl 10y ago
  Deep Learning is heavily used in Kaggle competitions when there is image data. Even in non-image data competitions, a deep neural network will often be one of the models chosen to ensemble. They generally perform slightly worse than XGBoost models, but have the advantage that they often aren't closey correlated, which helps fighting overfitting when tuning ensembling hyperparamters.
  - selectron 10y ago
    For image competitions you are right. Neural networks are often in winning teams ensembles, but they require a lot more work than something like xgboost (gradient-boosted decision trees). For a dataset that isn't image processing or NLP, xgboost is in general much more widely used than neural nets. Neural nets suffer from the amount of computing resources and knowledge needed to apply them, though given infinite knowledge and computing power they are probably on par with or better than xgboost. And if you need to analyze an image they are great.
agibsonccc 10y ago
To expand on this a bit: You need a baseline feature representation for structure data for "traditional" problems. What it DOES is shifts the problem to layer architectures which is a hard problem in and of itself: What vonnik is talking about and what's already been addressed in a sibling comment was images. "Features" are the pixels which handles generating the "baseline representation" needed for feature engineering. Another thing of note here is it just highlights that ETL is important. Eg: image sizes, window lengths for time series etc. Normalization is also still hugely important for many neural network problems. Smerity says it best: http://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html http://smerity.com/articles/2016/architectures_are_the_new_f...

by vonnik 10y ago

Eridrus 10y ago

I think the whole "you don't need to engineer features" thing can be a bit oversold, in that you don't need to build explicit feature extractors, but you do still need to design your architecture to fit your problem domain. It's more general than traditional feature extraction, but if you keep working on a single problem for long enough you may find that while deep learning can be a great starting point, you eventually want to add more hand-coded features if you really have some insight that the neural net isn't fully capturing. I'm personally quite interested to see if we find meaningful ways of constructing differentiable feature extractors that we can train conjointly with a neural net.

bhntr3 10y ago

I think the post represents the typical data scientist's approach to a machine learning problem pretty well. Deep learning is exciting but, for practical problems, most people aren't using it yet. XGBoost is still dominant in Kaggle competitions. Lots of companies with lots of data still use simple single machine models. So, I think it's a little misleading to say that deep learning eliminates the need for feature engineering. If we're going to provide business value to a company, we're often dealing with somewhat structured data and not "unstructured blobs of pixels or blobs of text". There are several simple linear models with highly engineered features in production where I work that resist attempts to replace them with more clever models and less feature engineering. Deep learning is awesome and there may be a time when it solves every problem. But let's not oversell it. For now, if someone wants to do machine learning and get paid for it, they're more likely to see their colleagues using the techniques in this article than training deep multi-layer neural nets.

nl 10y ago
Deep Learning is heavily used in Kaggle competitions when there is image data. Even in non-image data competitions, a deep neural network will often be one of the models chosen to ensemble. They generally perform slightly worse than XGBoost models, but have the advantage that they often aren't closey correlated, which helps fighting overfitting when tuning ensembling hyperparamters.
- selectron 10y ago
  For image competitions you are right. Neural networks are often in winning teams ensembles, but they require a lot more work than something like xgboost (gradient-boosted decision trees). For a dataset that isn't image processing or NLP, xgboost is in general much more widely used than neural nets. Neural nets suffer from the amount of computing resources and knowledge needed to apply them, though given infinite knowledge and computing power they are probably on par with or better than xgboost. And if you need to analyze an image they are great.

agibsonccc 10y ago

To expand on this a bit: You need a baseline feature representation for structure data for "traditional" problems. What it DOES is shifts the problem to layer architectures which is a hard problem in and of itself: What vonnik is talking about and what's already been addressed in a sibling comment was images. "Features" are the pixels which handles generating the "baseline representation" needed for feature engineering. Another thing of note here is it just highlights that ETL is important. Eg: image sizes, window lengths for time series etc. Normalization is also still hugely important for many neural network problems. Smerity says it best: http://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html http://smerity.com/articles/2016/architectures_are_the_new_f...