15 ms·
If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canoni
by astrophysician 6y ago
If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them.
In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning curve, especially if you're new to the game, but the reward is great.
All of these methods have their place. SVM's have their place, but also aren't great for probability calibration and non-linear SVM's like every single kernel method can scale absolutely terribly. Neural networks have their place, sometimes as a component of a larger statistical model, sometimes as a feature selector, sometimes in and of themselves. They're also very often the wrong choice for a problem.
Don't fall into the beginner trap: sometimes people tend to mistake 'what is the hottest research topic' for 'what is the right solution to my problem given my constraints, (data limitations, time limitations, skill limitations, etc.)'. Be realistic, don't use magical thinking, and have a strong basis in statistics to weed out the beautiful non-bullshit from the bullshit that is frustratingly prevalent (everyone and their mother is an ML expert today).
EDIT: I want to also clarify: I don't mean to suggest the author is new to ML, I just mean this as general advice for anyone coming here who is new to DS/ML. The article looks great!
- rangerranvir 6y agoThanks for the advice. Will definitely try to follow that. I was trying to learn basics of statistics and went through most of the intro to statistical learning. Will complete the rest in few days. I am more of a book person, if you have any other resource for probabilistic graphical models, please share here.
- astrophysician 6y agoNice! There are many books that cover this, even the docs for Pyro/other libraries are useful, it just depends on your preference for how material is presented + your background. Bishop's "Pattern Recognition and Machine Learning" has a chapter on PGM's that's free online: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/05/Bishop-PRML-sample.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... Murphy's "Machine Learning: A Probabilistic Perspective" is another behemoth that covers this stuff, but it's really just your preference. I say "aspire" because (1) depending on your background, it will likely be something that takes awhile to internalize and really understand, and you will probably realize many times over that you thought you understood something that you actually didn't (2) by learning PGM's, you learn a lot of Bayesian statistics as a side effect, hence why even learning a little bit about them is rewarding. Once you learn a bit, I would use Pyro/other libraries and try to actually build PGM's for toy problems (or non-toy problems too..) because (1) it will force you to admit to yourself that you don't understand something, (2) the documentation for a lot of these libraries is also useful learning material, and (3) you will see once you learn these libraries that it is fairly easy to do something that would be astoundingly complex if you were to try and do it by hand. You can basically build most standard ML algorithms as a PGM, so e.g. you can try to do logistic regression as a PGM and compare the results to scikit-learn.
- tesseract2 6y agoThanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?
- rangerranvir 6y ago+1
- uoaei 6y agoIntroduction to Statistical Learning https://faculty.marshall.usc.edu/gareth-james/ISL/ https://faculty.marshall.usc.edu/gareth-james/ISL/ Elements of Statistical Learning https://web.stanford.edu/~hastie/ElemStatLearn/ https://web.stanford.edu/~hastie/ElemStatLearn/ Machine Learning: A Probabilistic Perspective https://mitpress.mit.edu/books/machine-learning-1 https://mitpress.mit.edu/books/machine-learning-1
- rangerranvir 6y agoThanks a ton for these. Added this to things I know that I don't know list. ;)
- kmundnic 6y ago"Machine Learning: a Probabilistic Perspective" is more an encyclopedia of algorithms I would say, and it has lots of typos. I personally would not recommend it (except for the amount of algorithms that it covers, many of which are usually not found in other books).
- rangerranvir 6y agoThanks for early warning. Will have to keep that in mind.
- antipaul 6y agoAre those really the best starts for "Bayesian statistics"? Especially the first 2 are rather the standard "intro to ML textbooks", with a frequentist focus (ISL may even have zero Bayesian stuff - Naive Bayes is not "Bayesian" – while ESL still has maybe 10% bayesian content if that). Instead, I would suggest the following for learning Bayesian methods, especially given the HN crowd: https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers https://github.com/CamDavidsonPilon/Probabilistic-Programmin...
- uoaei 6y agoGreat comments. I heartily agree and support the statement about probabilistic graphical models. Just to add a couple more facets to this perspective: 'State of the art' does not always mean 'best for your task', and in fact lately depending on your field SOTA sometimes simply means 'unaffordable' for anyone whose budget is under 1 million dollars. Try linear methods first. Ensembles of decent models are usually good models. The point above about probability calibration can be at least somewhat mitigated by using ensemble averages. Don't just assume "the $MODEL will figure it out" if you give it shitloads of degrees of freedom. Machine learning efficiency all comes down to efficiency of representation, and feature engineering can achieve huge payoffs if/when you incorporate domain knowledge and expertise. Once you gain a perspective into the "universality" of statistical methods, optimization, and Bayesian probability theory, your work will become a lot easier to reason about. As an example, try to see if you can explain why least-squares fit results from the assumption that model residuals are normally distributed (and what connections this may have to statistical physics!).
- rangerranvir 6y agoGreat Points. Really appreciated. Will have to put extra effort to learn about the feature engineering part of the problem. Also, if you know a few things about the data it becomes a little easier to explain what your model is doing and why it is producing those results. Found a good resource which explained the trust component: https://arxiv.org/pdf/1602.04938.pdf https://arxiv.org/pdf/1602.04938.pdf
- ska 6y ago> Try linear methods first This bears repeating.
- disgruntledphd2 6y agoI really find it hard to believe that not everyone starts with a linear model. But apparently that's not hip any more.
- antipaul 6y agoCan you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task? Want to whet my appetite for your suggestion.
- bencw 6y agoA common type of example involves relatively small or uninformative datasets. Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads. The above example is contrived, but makes more sense in the case of language modelling. Since a bag-of-words vector, containing say counts of words seen in a document, is typically sparse (most documents only contain a limited portion of the full vocabulary), a frequentist estimate of word probability will say that certain words can never occur, just because it's never seen them. The Bayesian estimate will still assign some non-zero chance of seeing that word. Practically speaking, this leads to the idea of "smoothing" in tf-idf (text-frequency-inverse-document-frequency) vectors, by adding 1 to document frequencies. You don't need Bayesian statistics to do this, but maybe you never would have thought of it otherwise!
- astrofinch 6y ago>Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads. Not quite. If you have a uniform prior, there will be no difference between MAP and MLE. >From the vantage point of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters. https://en.wikipedia.org/wiki/Maximum_likelihood_estimation https://en.wikipedia.org/wiki/Maximum_likelihood_estimation More discussion here: https://stats.stackexchange.com/questions/64259/how-does-a-uniform-prior-lead-to-the-same-estimates-from-maximum-likelihood-and https://stats.stackexchange.com/questions/64259/how-does-a-u...
- rjvir 6y agoDifferent people learn in different ways, but personally I’ve had more success with the opposite approach, ie “top-down”. As in, rather than learning in depth all the low level parts then finally putting it together at the end, start with a surface high-level understanding of a working prototype then expand into the details of how everything works inside. In the case of ML, this could mean starting with a 5 line SciKit-learn prototype of a random forest model, seeing some working predictions, then expanding knowledge from there - what data is going in and what is coming out? What’s a classifier? What’s a decision tree? Etc
- antipaul 6y agoI support this learning method. Having a map of the key concepts, as well as visceral feel for them via code, will keep up the motivation. This would be in contrast to picking up one of the plethora of “ML” textbooks that mostly only describe the math behind all the algorithms. Which is not where you should begin, in my view (years of teaching experience). The use of such textbooks is as a reference to fill in details once your are curious about them. And more than anything, the best way to learn practical ML is to “apprentice” to some experienced practitioners or team who are willing to act as mentors.
- astrophysician 6y agoAgree — people should do what they want and have fun learning. Its just a suggestion that’s also colored by my own experience. I will argue that if you’re going for a job in data science it is a bit more of a serious suggestion since you need to be able to know how to answer statistical questions and understand your assumptions, and you really do need to understand Bayesian stats for that (nothing state of the art here, don’t worry if you don’t know what a PGM is, I just mean basics).
- rangerranvir 6y agoGreat Point! I do read article where someone builds such cool stuff every now and then to keep in mind what's coming. Anyway whatever works. Ultimate aim is to learn and have fun.
- bitL 6y agoStarting with PGMs would kill 99.9% of aspiring ML practitioners. Classes related to PGM at Stanford and MIT are considered to be some of the most difficult ones. I'd rather recommend to start with something they are enthusiastic about and once they become sufficiently advanced, to naturally learn (H)PGM.
- salty_biscuits 6y agoI think they said "aspire to", not start with. I would start with a naive bayes classifier, understand why it is naive and then go from there.
- astrophysician 6y agoExactly. I’m talking about how to orient yourself, PGMs are hard, sure but they don’t have to be outrageously hard. I would argue that if you do understand naive bayes (and it’s naivety) and you understand priors, then PGMs are just like rules to a game called “make a diagram of your posterior”. That’s not all there is obviously, and that’s kind of my point; you can do a lot with a little bit of knowledge, and then you can slowly climb that ladder for a long time and the more you learn, the more you can apply. Starting with an ad hoc approach (here are all of these classes in scikit learn with .fit() functions, lets just memorize their docstrings) isn’t “bad” per se, that knowledge is important, but it will not take you very deep, and you won’t be able to stray very far from those methods without being out of your depth and running into trouble.
- sjg007 6y agoPGMs are graphs mapped into probability theory and vice versa.
- mrslave 6y agoRequesting best book(s) on probability estimation: techniques, model accuracy, and strategies in applying them (e.g. markets, marketing, business operations)?
- deleted 6y ago[deleted]
- unishark 6y agoPersonally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estimation which is more of a hack to ML than an actual Bayesian method) and provide the big picture for almost everything. Meanwhile a strong basis in "deterministic methods", as an alternative way to spend that learning effort, has its own rewards. The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. For that matter a thorough understanding of SVM delves into convex optimization, an extremely powerful framework as well.
- astrofinch 6y ago>The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. The lore I've heard is that most new deep learning training algorithms (optimization algorithms) only work better on particular special cases, and it is hard to do better than the established algorithms in general. I'm also not sure why you're saying they're applicable beyond deep learning--how do you plan to train a PGM or SVM using Adam?
- astrophysician 6y agoAdam, RMSProp, etc are just flavors of gradient descent so they’re useful on anything from ResNet to logistic regression. There are more flavors like natural gradient that are more useful for smaller problems since they require a Hessian matrix, but gradient descent is gradient descent. We use Adam in production for logistic regression, not for any particular reason really, just happens to work.
- rangerranvir 6y agoRandom question, do you guys have tried linear regression in place of Adam and was there a significant change in the training time? And on what data you decide to move on to NN based model rather than using a simpler models like linear regression etc?
- sriram_malhar 6y agoCan you please suggest some resources for a math-comfortable beginner?
- jononor 6y agoDo you have any learning resources to recommend for Baysian ML? Especially interested in more applied stuff, and ideally temporal and spatial modeling.