4 ms·

I think the problem is that we don't really understand ML properly yet. Like picking hyperparamters - time and time again I've asked experts/trainers/colleague

I think the problem is that we don't really understand ML properly yet. Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". To me, that feels weird and worrying. Its like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need. This can lead us down the debugging black-hole TFA talks about since we appear to have zero-clue about why we chose something, so debugging something ultimately might just be "opps - we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! D'oh! Lets start training again!" It really grates me to think about this considering how much maths and proofs and algorithms get thrown at you when being taught ML, then to be told when it comes to actually doing something its all down to "intuition" (guessing). And yeah as others have said - data :-)

jps359 10y ago
Well, generally, we know why certain models work on certain types of data. But picking a proper model and hyperparameters requires some insight/knowledge about the properties of the data itself, which is the whole purpose of doing ML experiments in the first place (to gain knowledge/insight about the data)! However I do agree that we don't fully understand the area yet or the reach of its applications.
makmanalp 10y ago
> just try a load of stuff and pick the one that works best Well, yeah, that's easier to say than "read this entire book to understand how machine learning models work". The truth is that different models work with different success for different kinds of data. How many predictors do I have? How many observations vs how many predictors? How much data overall? Do I care about interpretability? Am I doing classification or regression? If doing regression, what is my tolerance for error? How much time am I willing to spend training and predicting? If doing classification, are my classes balanced or imbalanced? How are the classes spread? Do I care more about overall errors or false positives / false negatives (think of a cancer screening test - you might rather have false positives than miss something with false negatives). All of these factors, and more, go into play when selecting a model. There's other bits too. Selecting variables: including unneeded or highly correlated variables can actually worsen the result. Feature engineering: Turns out you might be able to process your variables in such a way that the algorithm picks up on the details you need much more easily. Cross validation: You don't want to "overfit" your training data, i.e. get a model that is super specific to your training data such that when encountering actual data points, it has worse accuracy because it's not general enough. Hyperparameter tuning: Usually a lot of these models have tuning parameters that you can tweak, that are hard to know what the values should be from the get go, and you have to try a bunch and look at the response curve of how the accuracy changes. So yeah, machine learning is not magic. Turns out there are different tools for different problems. We do have some models like Random Forests and SVMs that work fairly well out of the box on a wide variety of problems, and some kinds of neural networks also do well but often need more data and processing time to get decent results. It's all a tradeoff :)
narrator 10y ago
When I was taking the MIT analog circuits class, they were showing the differential equation for RLC circuits and the teacher explained that these were figured out via "guesswork". Apparently these equations are not solveable using a step by step simple transformation. If you plug them into Mathematica, it won't know how to solve them. Some really smart mathematicIan figured them out intuitively. This is what machine learning to me feels like. We just hook this stuff together and some really smart guy with a fractional Ramanujen worth of natural intuition says it will work and it does, especially for the really complex image recognition models. Have you guys looked at Google's "inception" architecture? It's a huge rube Goldberg machine of many many layers and it does work, but there's not a lot of reasoning about why it was designed the way it was.
scoot 10y ago
> the answer is always along the lines of "just try a load of stuff and pick the one that works best" Forgive the naive question, but why couldn't ML figure out its own best "stuff"?
- Silfen 10y ago
  This is pretty much what boosting is. Although we can ask the same question of the boosting algorithm, I guess.
- chongli 10y ago
  Because then you've got to figure out how to tell the machine what best means and that's what you were setting out to do in the first place.
highd 10y ago
To be clear your complaints only described a subset of ML - neural network approaches in particular.
giardini 10y ago
mattlondon> "And the answer is always along the lines of "just try a load of stuff and pick the one that works best"." Sounds to me like an opportunity to apply automation - Monte Carlo is your friend and machine cycles are cheap! Most software development and ML development are diametrically opposed. That's why we have statisticians. Most software developers like the control that programming gives them - that's why they are programmers. They prefer developing systems that can be programmed (trained) quickly, consistently and with rapid feedback using computer languages. ML in contrast, is sort of like the claw crane arcade game at Walmart, you remember, the one where you insert 50 cents and then get to "control" the movement of a scoop for 10 seconds and then it drops onto a pile of possible prizes and, when you raise the crane, it drops the prize back onto the pile. https://en.wikipedia.org/wiki/Claw_crane https://en.wikipedia.org/wiki/Claw_crane If you're very lucky, you get a stuffed bunny or duck for your kid. Usually your kid goes away disappointed. There's something there but it's just beyond your ability to find it in 5 minutes. That's what ML looks and feels like but 50X worse. The ML training software has 50-100 knobs(you get to pick!) and the object of your desire lives in a multi-dimensional hyperspace. Find out how to use the knobs to reach your boss' goal. This is imo NOT a task for a software developer. And you don't even get a fuzzy bunny for your efforts. Yes, the claw cranes are rigged: http://kotaku.com/5929888/why-yes-those-claw-machines-are-rigged-says-arcade-operator http://kotaku.com/5929888/why-yes-those-claw-machines-are-ri... And if you think that's bad, consider that much of AI-related ML is like trying to succeed with a claw crane designed by God (or maybe the Devil). It pays to choose your challenges carefully.
BickNowstrom 10y ago
Good rule-of-thumb to always add dropout. Very hard to lower the performance this way. Yes, we have to try a load of stuff, especially in deep learning, where the old feature engineering seems to be replaced with architecture engineering [1]. Yet, the stuff to try is usually well-known [2]. You should not be trying stuff completely at random (unless you do a random gridsearch for hyperparams [3]). > we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! Hinton advises to start with a wide net, ensure that the net is able to learn the function, then add dropout, smaller layers, and/or regularization to reduce the overfit. This should avoid scenario's like the above. Wanting to have the perfect model for a specific problem mathematically calculated may be an unrealistic demand. If we view machine learning as compression (using the shortest program possible to describe a function/distribution), per Kolmogorov Complexity, we can't compute this shortest program, and if we happen to find it, we can't know we did. The maths is for theory. Intuition is for the practical application. See intuition more as guided guessing, instead of random guessing: with experience you should know what are the flaws and benefits of the different models and architectures for particular problems. You can limit your search to sane narrow ranges. [1] https://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html https://smerity.com/articles/2016/architectures_are_the_new_... [2] Finally, suppose you want to train an LDNN. Rumor has it that it’s very difficult to do so, that it is “black magic” that requires years of experience. And while it is true that experience helps quite a bit, the amount of “trickery” is surprisingly limited ---- one needs be on the lookout for only a small number well-known pitfalls. - http://yyue.blogspot.com.br/2015/01/a-brief-overview-of-deep-learning.html http://yyue.blogspot.com.br/2015/01/a-brief-overview-of-deep... [3] http://www.jmlr.org/papers/v13/bergstra12a.html http://www.jmlr.org/papers/v13/bergstra12a.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html
cshep 10y ago
Another complication, especially for newcomers, is the boundary at which to use certain algorithms. Questions like 'Can/should I just use an SVM over deep learning?' are also suspect to 'guessing'. Mind, it's this explorative process that makes ML so intriguing too.

jps359 10y ago

Well, generally, we know why certain models work on certain types of data. But picking a proper model and hyperparameters requires some insight/knowledge about the properties of the data itself, which is the whole purpose of doing ML experiments in the first place (to gain knowledge/insight about the data)! However I do agree that we don't fully understand the area yet or the reach of its applications.

makmanalp 10y ago

> just try a load of stuff and pick the one that works best Well, yeah, that's easier to say than "read this entire book to understand how machine learning models work". The truth is that different models work with different success for different kinds of data. How many predictors do I have? How many observations vs how many predictors? How much data overall? Do I care about interpretability? Am I doing classification or regression? If doing regression, what is my tolerance for error? How much time am I willing to spend training and predicting? If doing classification, are my classes balanced or imbalanced? How are the classes spread? Do I care more about overall errors or false positives / false negatives (think of a cancer screening test - you might rather have false positives than miss something with false negatives). All of these factors, and more, go into play when selecting a model. There's other bits too. Selecting variables: including unneeded or highly correlated variables can actually worsen the result. Feature engineering: Turns out you might be able to process your variables in such a way that the algorithm picks up on the details you need much more easily. Cross validation: You don't want to "overfit" your training data, i.e. get a model that is super specific to your training data such that when encountering actual data points, it has worse accuracy because it's not general enough. Hyperparameter tuning: Usually a lot of these models have tuning parameters that you can tweak, that are hard to know what the values should be from the get go, and you have to try a bunch and look at the response curve of how the accuracy changes. So yeah, machine learning is not magic. Turns out there are different tools for different problems. We do have some models like Random Forests and SVMs that work fairly well out of the box on a wide variety of problems, and some kinds of neural networks also do well but often need more data and processing time to get decent results. It's all a tradeoff :)

narrator 10y ago

When I was taking the MIT analog circuits class, they were showing the differential equation for RLC circuits and the teacher explained that these were figured out via "guesswork". Apparently these equations are not solveable using a step by step simple transformation. If you plug them into Mathematica, it won't know how to solve them. Some really smart mathematicIan figured them out intuitively. This is what machine learning to me feels like. We just hook this stuff together and some really smart guy with a fractional Ramanujen worth of natural intuition says it will work and it does, especially for the really complex image recognition models. Have you guys looked at Google's "inception" architecture? It's a huge rube Goldberg machine of many many layers and it does work, but there's not a lot of reasoning about why it was designed the way it was.

scoot 10y ago

> the answer is always along the lines of "just try a load of stuff and pick the one that works best" Forgive the naive question, but why couldn't ML figure out its own best "stuff"?

Silfen 10y ago
This is pretty much what boosting is. Although we can ask the same question of the boosting algorithm, I guess.
chongli 10y ago
Because then you've got to figure out how to tell the machine what best means and that's what you were setting out to do in the first place.

highd 10y ago

To be clear your complaints only described a subset of ML - neural network approaches in particular.

giardini 10y ago

mattlondon> "And the answer is always along the lines of "just try a load of stuff and pick the one that works best"." Sounds to me like an opportunity to apply automation - Monte Carlo is your friend and machine cycles are cheap! Most software development and ML development are diametrically opposed. That's why we have statisticians. Most software developers like the control that programming gives them - that's why they are programmers. They prefer developing systems that can be programmed (trained) quickly, consistently and with rapid feedback using computer languages. ML in contrast, is sort of like the claw crane arcade game at Walmart, you remember, the one where you insert 50 cents and then get to "control" the movement of a scoop for 10 seconds and then it drops onto a pile of possible prizes and, when you raise the crane, it drops the prize back onto the pile. https://en.wikipedia.org/wiki/Claw_crane https://en.wikipedia.org/wiki/Claw_crane If you're very lucky, you get a stuffed bunny or duck for your kid. Usually your kid goes away disappointed. There's something there but it's just beyond your ability to find it in 5 minutes. That's what ML looks and feels like but 50X worse. The ML training software has 50-100 knobs(you get to pick!) and the object of your desire lives in a multi-dimensional hyperspace. Find out how to use the knobs to reach your boss' goal. This is imo NOT a task for a software developer. And you don't even get a fuzzy bunny for your efforts. Yes, the claw cranes are rigged: http://kotaku.com/5929888/why-yes-those-claw-machines-are-rigged-says-arcade-operator http://kotaku.com/5929888/why-yes-those-claw-machines-are-ri... And if you think that's bad, consider that much of AI-related ML is like trying to succeed with a claw crane designed by God (or maybe the Devil). It pays to choose your challenges carefully.

BickNowstrom 10y ago

Good rule-of-thumb to always add dropout. Very hard to lower the performance this way. Yes, we have to try a load of stuff, especially in deep learning, where the old feature engineering seems to be replaced with architecture engineering [1]. Yet, the stuff to try is usually well-known [2]. You should not be trying stuff completely at random (unless you do a random gridsearch for hyperparams [3]). > we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! Hinton advises to start with a wide net, ensure that the net is able to learn the function, then add dropout, smaller layers, and/or regularization to reduce the overfit. This should avoid scenario's like the above. Wanting to have the perfect model for a specific problem mathematically calculated may be an unrealistic demand. If we view machine learning as compression (using the shortest program possible to describe a function/distribution), per Kolmogorov Complexity, we can't compute this shortest program, and if we happen to find it, we can't know we did. The maths is for theory. Intuition is for the practical application. See intuition more as guided guessing, instead of random guessing: with experience you should know what are the flaws and benefits of the different models and architectures for particular problems. You can limit your search to sane narrow ranges. [1] https://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html https://smerity.com/articles/2016/architectures_are_the_new_... [2] Finally, suppose you want to train an LDNN. Rumor has it that it’s very difficult to do so, that it is “black magic” that requires years of experience. And while it is true that experience helps quite a bit, the amount of “trickery” is surprisingly limited ---- one needs be on the lookout for only a small number well-known pitfalls. - http://yyue.blogspot.com.br/2015/01/a-brief-overview-of-deep-learning.html http://yyue.blogspot.com.br/2015/01/a-brief-overview-of-deep... [3] http://www.jmlr.org/papers/v13/bergstra12a.html http://www.jmlr.org/papers/v13/bergstra12a.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html

cshep 10y ago

Another complication, especially for newcomers, is the boundary at which to use certain algorithms. Questions like 'Can/should I just use an SVM over deep learning?' are also suspect to 'guessing'. Mind, it's this explorative process that makes ML so intriguing too.