Good rule-of-thumb to always add dropout. Very hard to lower the performance this way. Yes, we have to try a load of stuff, especially in deep learning, where

Good rule-of-thumb to always add dropout. Very hard to lower the performance this way. Yes, we have to try a load of stuff, especially in deep learning, where the old feature engineering seems to be replaced with architecture engineering [1]. Yet, the stuff to try is usually well-known [2]. You should not be trying stuff completely at random (unless you do a random gridsearch for hyperparams [3]). > we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! Hinton advises to start with a wide net, ensure that the net is able to learn the function, then add dropout, smaller layers, and/or regularization to reduce the overfit. This should avoid scenario's like the above. Wanting to have the perfect model for a specific problem mathematically calculated may be an unrealistic demand. If we view machine learning as compression (using the shortest program possible to describe a function/distribution), per Kolmogorov Complexity, we can't compute this shortest program, and if we happen to find it, we can't know we did. The maths is for theory. Intuition is for the practical application. See intuition more as guided guessing, instead of random guessing: with experience you should know what are the flaws and benefits of the different models and architectures for particular problems. You can limit your search to sane narrow ranges. [1] https://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html https://smerity.com/articles/2016/architectures_are_the_new_... [2] Finally, suppose you want to train an LDNN. Rumor has it that it’s very difficult to do so, that it is “black magic” that requires years of experience. And while it is true that experience helps quite a bit, the amount of “trickery” is surprisingly limited ---- one needs be on the lookout for only a small number well-known pitfalls. - http://yyue.blogspot.com.br/2015/01/a-brief-overview-of-deep-learning.html http://yyue.blogspot.com.br/2015/01/a-brief-overview-of-deep... [3] http://www.jmlr.org/papers/v13/bergstra12a.html http://www.jmlr.org/papers/v13/bergstra12a.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html