5 ms·
Understanding Machine Learning: From Theory to Algorithms
- TechDebtDevin 2y agoAnyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer. To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are already an expert as it will give you great ways to teach and communicate complex ideas in ML. [0]: https://www.goodreads.com/book/show/75622146-the-statquest-illustrated-guide-to-machine-learning https://www.goodreads.com/book/show/75622146-the-statquest-i...
- joshdavham 2y agoI haven't read that book, but I can personally attest to Josh Starmer's StatQuest Youtube channel[1] being awesome! I used his lessons as a supplement to my studies when I was studying statistics in uni. [1]: https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw
- gavinray 2y agoThis is the 2nd or 3rd time in the last few weeks I've seen this person recommended. Must be something to that.
- j_bum 2y agoHe’s great. I learned a ton from him when I was starting my computational biology studies as a grad student.
- drivers99 2y agoI thought I was having some deja vu. I had to keep checking the timestamps. The same person recommended the book 18 days ago: https://news.ycombinator.com/item?id=43390896 https://news.ycombinator.com/item?id=43390896 and then a different person also recommended the author's YouTube channel. I also bought the book since then. I guess I better get started reading it now. :)
- mirekrusin 2y agoIs there a chapter there on optimising sells through artificial activity? ;)
- OJFord 2y agoNot just recommending it again, but exactly the same comment.
- heavymetalpoizn 2y ago[dead]
- RealityVoid 2y agoI have it in my bookshelf! I bought it on a whim, used, along with other CS books, but didn't think it's that good! I will try reading it. Thanks.
- kenjackson 2y agoI would've thought that NN and ML would be taught together. Does he assume with the NN book that you already have a certain level of ML understanding?
- m11a 2y agoMost ML is disjoint from the current NN trends, IMO. Compare Bishop's PRML to his Deep Learning textbook. First couple chapters are copy+paste preliminaries (probability, statistics, Gaussians, other maths background), and then they completely diverge. I'm not sure how useful classical ML is for understanding NNs.
- xmprt 2y agoThat's fair. My understanding is that NN and ML are similar insofar as they are both about minimizing a loss value (like negative log likelihood). And then the methods of doing that are very different and once you get even more advanced, NN concepts feel like a completely different universe.
- grep_it 2y agoThanks for the recommendation. Purchased both them!
- sn9 2y agoThere's also Math Academy's Math for Machine Learning.
- mattfrommars 2y agoI'm just curious for folks who have read through the material OP suggested as well as, the book linked in this HN thread, are your guys primary motivation to understand and fill in that curiosity part of your head vs making a career out of this? Is it reasonable to think that if one grinds to the book suggested here and background in web/dev SWE, one can break into ML/AI role?
- randomNumber7 2y agoMost ML/AI roles have requirements for a strong mathematical background (at least what I have seen in germany). If you can show off some skills I still wouldnt completely rule it out. Reading a single book cover to cover wont cut it though imo.
- GPerson 2y agoIf you have an undergraduate’s understanding of calculus and linear algebra, you’re as or more advanced than the legion of ML PhD candidates I see graduating all the time. A field like that is running on hype, and has no quality control at all. I’ve seen people get hired into Ivy League tenure track jobs without knowing how linear algebra really works.
- randomNumber7 2y agoI know ML PhD candidates with a very strong understanding of mathematics and statistic, but it might depend on where you are.
- maCDzP 2y agoDidn’t know he had written a book. His YouTube channel is awesome.
- dkga 2y agoI haven‘t yet read the book but his Youtube channel is always my first go-to place for ideas on how to communicate these concepts easily. My work involves using ML in econometric analyses and most economists do not intuitively understand ML.
- fancyfredbot 2y agoI love StatsQuest and own this book which I like a lot. However I could not recommend it as a way of ML beyond a surface level. It's a little bit outdated although his NN book, which I've not read, may remedy this.
- cubefox 2y agoI have read parts of it years ago. As far as I remember, this is very theoretical (lots of statistical learning theory, including some IMHO mistaken treatment of Vapnik's theory of structural risk minimization), with strong focus on theory and basicasically zero focus on applications. Which would be completely outdated by now anyway, as the book is from 2014, an eternity in AI. I don't think many people will want to read it today. As far as I know, mathematical theories like SLT have been of little use for the invention of transformers or for explaining why neural networks don't overfit despite large VC dimension. Edit: I think the title "From theory to machine learning" sums up what was wrong with this theory-first approach. Basically, people with interest in math but with no interest in software engineering got interested in ML and invented various abstract "learning theories", e.g. statistical learning theory (SLT). Which had very little to do with what you can do in practice. Meanwhile, engineers ignored those theories and got their hands dirty on actual neural network implementations while trying to figure out how their performance can be improved, which led to things like CNNs and later transformers. I remember Vapnik (the V in VC dimension) complaining in the preface to one of his books about the prevalent (alleged) extremism of focussing on practice only while ignoring all those beautiful math theories. As far as I know, it has now turned out that these theories just were far too weak to explain the actual complexity of approaches that do work in practice. It has clearly turned out that machine learning is a branch of engineering, not a branch of mathematics or theoretical computer science. The title of this book encapsulates the mistaken hope that first people will learn those abstract learning theories, they get inspired, and promptly invent new algorithms. But that's not what happened. SLT is barely able to model supervised learning, let alone reinforcement learning or self-supervised learning. As I mentioned, they can't even explain why neural networks are robust to overfitting. Other learning theories (like computational/algorithmic learning theory, or fantasy stuff like Solomonoff induction / Kolmogorov complexity) are even more detached from reality.
- lamename 2y agoI watched a discussion the other day on this "NNs don't overfit point". I realize yes certain aspects are surprising, and in many cases with the right size and diversity in a dataset scaling laws prevail, but my experience with real datasets training from scratch (not fine tuning pretrained models), and impression has always been that NNs definitely can overfit if you don't have large quantities of data. My gut assumption is that original theories were not demonstrated to be true in certain circumstances (i.e. certain dataset characteristics), but that's never mentioned in shorthand these days when data sets size is often assumed to be huge. (Before anyone laughs this off, this is still an actual problem in the real world for non-FAANG companies who have niche problems or cannot use open-but-non-commercial datasets. Not everything can be solved with foundational/frontier models.) Please point me to these papers because I'm still learning.
- janis1234 2y agoBook is 10 years old, isn't it outdated?
- 0cf8612b2e1e 2y agoHave not read the book, but only deep learning has had such wild advancement that a decade would change anything. The fundamentals of ML training/testing, variance/bias, etc are the same. The classical algorithms still have their place. The only modern advancement which might not be present would be XGBoost style forests.
- TechDebtDevin 2y agoMachine Learning concepts have been around forever, they just used to call them statistics ;0
- antegamisou 2y agoNope, and AIMA/PRML/ESL are still king! Apart from these 3 you literally need nothing else for the very fundamentals and even advanced topics.
- raegis 2y agoThis is one of the most acronym heavy discussions I've ever seen. I searched "AIMA/PRML/ESL" to find the books, and the first result is a Reddit thread with most upvoted comment "Can we use the names of the books instead of all acronyms, not everyone knows them lol".
- antegamisou 2y agoYou're right. AIMA is Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig. PRM is Pattern Recognition and Machine Learning by Christopher Bishop. ESL is Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman.
- nyrikki 2y ago
- pajamasam 2y agoI would recommend https://udlbook.github.io/udlbook/ https://udlbook.github.io/udlbook/ instead if you're looking to learn about modern generative AI.
- smath 2y ago+1 for Simon prince’s UDL book. Very clearly written
- miltava 2y agoThanks for the recommendation. Have you looked at Bishop’s Deep learning book (https://www.bishopbook.com/ https://www.bishopbook.com/)? How would you compare both? Thanks again
- m11a 2y agoYou'll be happy with either. Bishop's approach is historically more mathematical (cf his 2006 PRML text), and you see that in the preliminaries chapters of Deep Learning, but there's less of this as the book goes on. I've read chapters from both. Much overlaps, but sometimes one book or the other explains a concept better or provides different perspectives or details.
- johnsutor 2y agohttps://bloomberg.github.io/foml/#home https://bloomberg.github.io/foml/#home This course is my personal favorite.
- joshdavham 2y agoWhat other books do people recommend?
- RamblingCTO 2y agoI really liked Peter Flach - Machine Learning. I think it's really well versed and not as aged as AIMA
- amit_m 2y agoAn Introduction to Statistical Learning: https://www.statlearning.com https://www.statlearning.com It is a free, easy introductory text by leading researchers that covers all the classics and includes many "Lab" sections with code. It has a chapter on deep learning but doesn't cover any of the recent advances. You will need other sources for that.
- deleted 2y ago[deleted]
- rottc0dd 2y agoFrom my other comment elsewhere. These resources helped me understand the topics better. If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1]. It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics) There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3]. There is also book for excellent caltech course[4]. Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5]. [1] https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4rOpr_A7B9SriE_iZmkanvUg https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... [2] https://chrispiech.github.io/probabilityForComputerScientists/en/ https://chrispiech.github.io/probabilityForComputerScientist... [3] https://work.caltech.edu/telecourse https://work.caltech.edu/telecourse [4] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/1600490069 https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/... [5] https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
- utopcell 2y agoThis is from 2014. Is it really relevant anymore?
- jlcases 2y agoThe biggest challenge with ML models isn't the algorithm but the organization of contextual knowledge. In my experience, hierarchical structuring of documentation significantly improves results, especially when working with LLMs.
- fithisux 2y agoI personally like Cosma Shalizi's book
- revskill 2y agoWhy not from algorithm to theory ?
- chriskanan 2y agoThis is my favorite introductory machine learning theory (statistical learning theory) book, which is far more accessible than many others.
- spacephysics 2y agoIs there utility for a software engineer to understand ML concepts to a deep degree if they don’t perform research? Trying to gauge where I should focus learning for my career (which i don’t plan to do research in) Roughly I see a gap in businesses needing AI/ML implemented, but outside some debugging, would it be worthwhile to develop a model from scratch or would some off the shelf model for use case X, tuned a bit, likely fit most use cases for a standard business?