7 ms·

Book: Mathematics for Machine Learning

465 points by nafizh 9y ago

breezest 9y ago
Nowadays, many books cover the elementary mathematics in machine learning. After I learnt these elementary topics, any good suggestions for computational learning theory?
- stochastic_monk 9y ago
  I recommend Shai Shalev-Schwartz and Shai Ben-David's Understanding Machine Learning: From Theory to Algorithms [0]. I've also used and found Mohri's Foundations of Machine Learning quite insightful [1]. Usually, between the two books, at least one's proof is easy to follow. Get both unless you're only getting one, in which case, get Shai Shalev-Schwartz's. [0] http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/ http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning... [1] https://mitpress.mit.edu/books/foundations-machine-learning https://mitpress.mit.edu/books/foundations-machine-learning
- make3 9y ago
  [Deleted]
  - stochastic_monk 9y ago
    He's specifically asking about learning theory which is a subfield of machine learning, along the lines of work you'll see at COLT, using concentration bounds, VC theory, and Rademacher complexity. PRML, Murphy, ESL, the Deep Learning book, and the RL introduction are more like what you'd see at ICML or ICLR.
    - make3 9y ago
      thanks, my bad.
  - foo101 9y ago
    Why delete the comment? Useful context for stochastic_monk's reply is missing now. You could always preserve the existing and possibly incorrect comment and append an "Update" or "Edit" section to the comment to override your earlier comment.
    - stochastic_monk 9y ago
      Essentially, he asked if the above poster had read Elements of Statistical Learning, Murphy's ML textbook, Bishop's PRML, Reinforcement Learning: An Introduction, and Ian Goodfellow's Deep Learning textbook. I simply clarified that the question was about computational learning theory, a subfield largely started by Leslie Valiant in the form of PAC (Probably Approximately Correct) learning. The difference in emphasis between the machine learning conferences I mentioned helps point out how practical machine learning (like ICML, matching PRML/ML/ESL) and feature extraction/representation learning (like ICLR, perhaps matching portions of both ICML and ICLR), while important, are not what the previous poster was asking about.
AndrewKemendo 9y ago
Not sure if the authors will read this or not but I beg of you, please put a table of notation in the forward. The Sutton and Barto Reinforcement Learning book did that for basically every notation that wasn't basic algebra and it's been extremely helpful. Just labeling things I had never seen before, like indicator functions, was extremely valuable. Especially for this kind of book that is introducing mathematics to people from a broad background - I think it's important to understand how much of an impediment not knowing notation is by sight. Trying to Google or search for notation is a nightmare.
- nafizh 9y ago
  They have a github repo. You can post this as an issue there, the authors are pretty responsive.
- mkl 9y ago
  Agreed. Another note in case the authors see this: using a third party URL shortener like tinyurl in a book seems fragile and crazy.
- bronxbomber92 9y ago
  The authors are open to feedback. They ask that you report it here: https://github.com/mml-book/mml-book.github.io/issues https://github.com/mml-book/mml-book.github.io/issues
  - chris_wot 9y ago
    I thought this was such a good idea I did it on the parent's behalf: https://github.com/mml-book/mml-book.github.io/issues/33 https://github.com/mml-book/mml-book.github.io/issues/33
- dfan 9y ago
  Yes, this is so useful when it is present. Whenever I read an even moderately mathematical paper I end up making one myself (including the meanings and types of all the variables).
burlesona 9y ago
It's not clear to me who the target audience is for this book. As someone with a graduate degree but not a mathematical background, I'm comfortable reading academic papers, but a lot of this feels over my head. So it reads to me like it's a quick refresher or reference for people who already learned this stuff at some point, rather than an introduction. But the preface says this is to be "a book on Mathematics for Machine Learning that motivates people to learn mathematical concepts." As a sidebar, it has always seemed to me that there is a giant gulf between truly beginner-friendly math books, which are aimed at children, and introductory math books aimed at adults. The latter almost always read like foreign language textbooks where you must first know the language before you can start, while the former are too elementary. I'd love to find "College level Math for English Majors" or something of the like, if anyone knows of such a book :)
- abhishekjha 9y ago
  At this point we might benefit a lot from a forum where mathematical concepts are discussed for a particular application in the most explicit ways and not just question and answers. We can then build up on that to further the mathematical concepts and also to rope in more people into proper Machine Learning domain. Is this objective a little too unrealistic to obtain?
  - jacobolus 9y ago
    That forum has existed for centuries, and is called a university.
    - foo101 9y ago
      How is that relevant? Many of us are not going to spend our entire lives in a university.
      - jacobolus 9y ago
        The general way to master a large pile of difficult abstractions is to study for years, ideally in a context focused on that study. This can be done independently of any expert feedback (e.g. by reading books, watching lecture videos, and doing a big pile of ungraded self-assigned homework problems), but it’s not easy or typically very efficient. For the majority of people who want to deeply learn mathematical topics but who are starting from a low level, the most efficient and effective advice for them is to go attend a university. An online discussion forum isn’t typically going to cut it, IMO. People won’t be at the same level, won’t be working on the same topics, won’t have the same goals and interests, the same level of commitment, coordinated schedules, etc. This has been tried many times, but I have not heard many successful reports. Online discussion forums are great for handling independent limited-scope questions–answers (including for current university students), and okay for coordinating research efforts among people who are already experts, but in general are not very effective for getting a group of people to diligently work through years of prerequisite material in an organized fashion. You shouldn’t spend your life in a university, but if you have a goal of learning advanced mathematics, you might benefit from spending a few years there. YMMV. As an alternative, if you can find an expert tutor/coach to meet with one-on-one on a regular basis, that is even better than a formal course. But that is expensive and hard to scale.
ReactForAll 9y ago
Here is my feed back, this book appears not to be useful and doesn't seem to provide any intuition to why we use any of the linear algebra to do work with machine learning ... it seems like a undergraduate book filled with descriptions of math. I find there needs to be a bridge to explain why we use vectors, how we model then relate it back to the math.
dsiegel2275 9y ago
For those interested in a similar resource, CMU offers a "Mathematics for Machine Learning" preparatory course each Fall semester. All of the videos from the 2017 edition are available to watch on YouTube. Syllabus: https://canvas.cmu.edu/courses/603/assignments/syllabus https://canvas.cmu.edu/courses/603/assignments/syllabus YouTube playlist: https://www.youtube.com/playlist?list=PL7y-1rk2cCsAqRtWoZ95z-GMcecVG5mzA https://www.youtube.com/playlist?list=PL7y-1rk2cCsAqRtWoZ95z...
- nafizh 9y ago
  Looks great. Unfortunately, the assignments are not available for public.
- dufferzafar 9y ago
  The syllabus looks good, but all videos have the same title. Is there a "topic" -> "video" mapping somewhere? Couldn't find it on the site.
  - dsiegel2275 9y ago
    There isn't, unfortunately.
theCricketer 9y ago
I've mentioned this on HN before but I think its still relevant to people interested in learning ML who feel they are behind on the math. If, like me, you can't sit thru lots of pages of mathematics text and instead prefer that a human explains it to you via videos that you can replay, here is a list of courses that take you from basic algebra and pre-calculus math all the way to the concepts you need to understand the principles behind the most advanced ML algorithms. All explained by very energetic people who are experts in their fields, and starting from the very basics. This covers calculus, linear algebra, probability, statistics, convex optimization and a math for ML course thrown in for the HN audience: (The first two are "MOOCs" recorded in the 1970s! probably the first ever recorded MOOC, even before the internet, and the lecturer is absolute gold) Calculus Revisited: Single Variable Calculus | MIT https://ocw.mit.edu/resources/res-18-006-calculus-revisited-.. https://ocw.mit.edu/resources/res-18-006-calculus-revisited-.... Calculus Revisited: Multivariable Calculus | MIT https://ocw.mit.edu/resources/res-18-007-calculus-revisited-.. https://ocw.mit.edu/resources/res-18-007-calculus-revisited-.... Complex Variables, Differential Equations, and Linear Algebra | MIT https://ocw.mit.edu/resources/res-18-008-calculus-revisited-.. https://ocw.mit.edu/resources/res-18-008-calculus-revisited-.... Linear Algebra | MIT - https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101.. https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101.... Introduction to Linear Dynamical Systems |Stanford https://see.stanford.edu/Course/EE263 https://see.stanford.edu/Course/EE263 Probability | Harvard https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6.. https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6.... Intermediate Statistics | CMU https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVW.. https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVW.... Convex Optimization I | Stanford https://see.stanford.edu/Course/EE364A https://see.stanford.edu/Course/EE364A Math Background for ML | CMU https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW.. https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW....
- nagarc 9y ago
  Many of the links are not working
  - thomanq 9y ago
    Links as per the original comment [0] Calculus Revisited: Single Variable Calculus | MIT https://ocw.mit.edu/resources/res-18-006-calculus-revisited-single-variable-calculus-fall-2010/index.htm https://ocw.mit.edu/resources/res-18-006-calculus-revisited-... Calculus Revisited: Multivariable Calculus | MIT https://ocw.mit.edu/resources/res-18-007-calculus-revisited-multivariable-calculus-fall-2011/index.htm https://ocw.mit.edu/resources/res-18-007-calculus-revisited-... Complex Variables, Differential Equations, and Linear Algebra | MIT https://ocw.mit.edu/resources/res-18-008-calculus-revisited-complex-variables-differential-equations-and-linear-algebra-fall-2011/index.htm https://ocw.mit.edu/resources/res-18-008-calculus-revisited-... Linear Algebra | MIT - https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD91010BC51F8 https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101... Introduction to Linear Dynamical Systems |Stanford https://see.stanford.edu/Course/EE263 https://see.stanford.edu/Course/EE263 Probability | Harvard https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6ht66KWxbzTIo https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6... Intermediate Statistics | CMU https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVWq-7wr8ffRtHF https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVW... Convex Optimization I | Stanford https://see.stanford.edu/Course/EE364A https://see.stanford.edu/Course/EE364A Math Background for ML | CMU https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMWUaBRuLBvPBCg https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW... [0] https://news.ycombinator.com/item?id=14581962 https://news.ycombinator.com/item?id=14581962
ghosthamlet 9y ago
This book is aslo Mathematics for Machine Learning: https://github.com/soulmachine/machine-learning-cheat-sheet https://github.com/soulmachine/machine-learning-cheat-sheet
vigdals 9y ago
TheNewBoston's youtube channel also has some nice math videos that might help someone. Not sure if its applicable here. https://www.youtube.com/watch?v=UxhMTi2bh7k&list=PLAF739DF5F2D9506C https://www.youtube.com/watch?v=UxhMTi2bh7k&list=PLAF739DF5F...

465 points by nafizh 9y ago

Nowadays, many books cover the elementary mathematics in machine learning. After I learnt these elementary topics, any good suggestions for computational learning theory?

stochastic_monk 9y ago
I recommend Shai Shalev-Schwartz and Shai Ben-David's Understanding Machine Learning: From Theory to Algorithms [0]. I've also used and found Mohri's Foundations of Machine Learning quite insightful [1]. Usually, between the two books, at least one's proof is easy to follow. Get both unless you're only getting one, in which case, get Shai Shalev-Schwartz's. [0] http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/ http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning... [1] https://mitpress.mit.edu/books/foundations-machine-learning https://mitpress.mit.edu/books/foundations-machine-learning
make3 9y ago
[Deleted]
- stochastic_monk 9y ago
  He's specifically asking about learning theory which is a subfield of machine learning, along the lines of work you'll see at COLT, using concentration bounds, VC theory, and Rademacher complexity. PRML, Murphy, ESL, the Deep Learning book, and the RL introduction are more like what you'd see at ICML or ICLR.
  - make3 9y ago
    thanks, my bad.
- foo101 9y ago
  Why delete the comment? Useful context for stochastic_monk's reply is missing now. You could always preserve the existing and possibly incorrect comment and append an "Update" or "Edit" section to the comment to override your earlier comment.
  - stochastic_monk 9y ago
    Essentially, he asked if the above poster had read Elements of Statistical Learning, Murphy's ML textbook, Bishop's PRML, Reinforcement Learning: An Introduction, and Ian Goodfellow's Deep Learning textbook. I simply clarified that the question was about computational learning theory, a subfield largely started by Leslie Valiant in the form of PAC (Probably Approximately Correct) learning. The difference in emphasis between the machine learning conferences I mentioned helps point out how practical machine learning (like ICML, matching PRML/ML/ESL) and feature extraction/representation learning (like ICLR, perhaps matching portions of both ICML and ICLR), while important, are not what the previous poster was asking about.

AndrewKemendo 9y ago

Not sure if the authors will read this or not but I beg of you, please put a table of notation in the forward. The Sutton and Barto Reinforcement Learning book did that for basically every notation that wasn't basic algebra and it's been extremely helpful. Just labeling things I had never seen before, like indicator functions, was extremely valuable. Especially for this kind of book that is introducing mathematics to people from a broad background - I think it's important to understand how much of an impediment not knowing notation is by sight. Trying to Google or search for notation is a nightmare.

nafizh 9y ago
They have a github repo. You can post this as an issue there, the authors are pretty responsive.
mkl 9y ago
Agreed. Another note in case the authors see this: using a third party URL shortener like tinyurl in a book seems fragile and crazy.
bronxbomber92 9y ago
The authors are open to feedback. They ask that you report it here: https://github.com/mml-book/mml-book.github.io/issues https://github.com/mml-book/mml-book.github.io/issues
- chris_wot 9y ago
  I thought this was such a good idea I did it on the parent's behalf: https://github.com/mml-book/mml-book.github.io/issues/33 https://github.com/mml-book/mml-book.github.io/issues/33
dfan 9y ago
Yes, this is so useful when it is present. Whenever I read an even moderately mathematical paper I end up making one myself (including the meanings and types of all the variables).

burlesona 9y ago

It's not clear to me who the target audience is for this book. As someone with a graduate degree but not a mathematical background, I'm comfortable reading academic papers, but a lot of this feels over my head. So it reads to me like it's a quick refresher or reference for people who already learned this stuff at some point, rather than an introduction. But the preface says this is to be "a book on Mathematics for Machine Learning that motivates people to learn mathematical concepts." As a sidebar, it has always seemed to me that there is a giant gulf between truly beginner-friendly math books, which are aimed at children, and introductory math books aimed at adults. The latter almost always read like foreign language textbooks where you must first know the language before you can start, while the former are too elementary. I'd love to find "College level Math for English Majors" or something of the like, if anyone knows of such a book :)

abhishekjha 9y ago
At this point we might benefit a lot from a forum where mathematical concepts are discussed for a particular application in the most explicit ways and not just question and answers. We can then build up on that to further the mathematical concepts and also to rope in more people into proper Machine Learning domain. Is this objective a little too unrealistic to obtain?
- jacobolus 9y ago
  That forum has existed for centuries, and is called a university.
  - foo101 9y ago
    How is that relevant? Many of us are not going to spend our entire lives in a university.
    - jacobolus 9y ago
      The general way to master a large pile of difficult abstractions is to study for years, ideally in a context focused on that study. This can be done independently of any expert feedback (e.g. by reading books, watching lecture videos, and doing a big pile of ungraded self-assigned homework problems), but it’s not easy or typically very efficient. For the majority of people who want to deeply learn mathematical topics but who are starting from a low level, the most efficient and effective advice for them is to go attend a university. An online discussion forum isn’t typically going to cut it, IMO. People won’t be at the same level, won’t be working on the same topics, won’t have the same goals and interests, the same level of commitment, coordinated schedules, etc. This has been tried many times, but I have not heard many successful reports. Online discussion forums are great for handling independent limited-scope questions–answers (including for current university students), and okay for coordinating research efforts among people who are already experts, but in general are not very effective for getting a group of people to diligently work through years of prerequisite material in an organized fashion. You shouldn’t spend your life in a university, but if you have a goal of learning advanced mathematics, you might benefit from spending a few years there. YMMV. As an alternative, if you can find an expert tutor/coach to meet with one-on-one on a regular basis, that is even better than a formal course. But that is expensive and hard to scale.

ReactForAll 9y ago

Here is my feed back, this book appears not to be useful and doesn't seem to provide any intuition to why we use any of the linear algebra to do work with machine learning ... it seems like a undergraduate book filled with descriptions of math. I find there needs to be a bridge to explain why we use vectors, how we model then relate it back to the math.

dsiegel2275 9y ago

For those interested in a similar resource, CMU offers a "Mathematics for Machine Learning" preparatory course each Fall semester. All of the videos from the 2017 edition are available to watch on YouTube. Syllabus: https://canvas.cmu.edu/courses/603/assignments/syllabus https://canvas.cmu.edu/courses/603/assignments/syllabus YouTube playlist: https://www.youtube.com/playlist?list=PL7y-1rk2cCsAqRtWoZ95z-GMcecVG5mzA https://www.youtube.com/playlist?list=PL7y-1rk2cCsAqRtWoZ95z...

nafizh 9y ago
Looks great. Unfortunately, the assignments are not available for public.
dufferzafar 9y ago
The syllabus looks good, but all videos have the same title. Is there a "topic" -> "video" mapping somewhere? Couldn't find it on the site.
- dsiegel2275 9y ago
  There isn't, unfortunately.

theCricketer 9y ago

I've mentioned this on HN before but I think its still relevant to people interested in learning ML who feel they are behind on the math. If, like me, you can't sit thru lots of pages of mathematics text and instead prefer that a human explains it to you via videos that you can replay, here is a list of courses that take you from basic algebra and pre-calculus math all the way to the concepts you need to understand the principles behind the most advanced ML algorithms. All explained by very energetic people who are experts in their fields, and starting from the very basics. This covers calculus, linear algebra, probability, statistics, convex optimization and a math for ML course thrown in for the HN audience: (The first two are "MOOCs" recorded in the 1970s! probably the first ever recorded MOOC, even before the internet, and the lecturer is absolute gold) Calculus Revisited: Single Variable Calculus | MIT https://ocw.mit.edu/resources/res-18-006-calculus-revisited-.. https://ocw.mit.edu/resources/res-18-006-calculus-revisited-.... Calculus Revisited: Multivariable Calculus | MIT https://ocw.mit.edu/resources/res-18-007-calculus-revisited-.. https://ocw.mit.edu/resources/res-18-007-calculus-revisited-.... Complex Variables, Differential Equations, and Linear Algebra | MIT https://ocw.mit.edu/resources/res-18-008-calculus-revisited-.. https://ocw.mit.edu/resources/res-18-008-calculus-revisited-.... Linear Algebra | MIT - https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101.. https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101.... Introduction to Linear Dynamical Systems |Stanford https://see.stanford.edu/Course/EE263 https://see.stanford.edu/Course/EE263 Probability | Harvard https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6.. https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6.... Intermediate Statistics | CMU https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVW.. https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVW.... Convex Optimization I | Stanford https://see.stanford.edu/Course/EE364A https://see.stanford.edu/Course/EE364A Math Background for ML | CMU https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW.. https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW....

nagarc 9y ago
Many of the links are not working
- thomanq 9y ago
  Links as per the original comment [0] Calculus Revisited: Single Variable Calculus | MIT https://ocw.mit.edu/resources/res-18-006-calculus-revisited-single-variable-calculus-fall-2010/index.htm https://ocw.mit.edu/resources/res-18-006-calculus-revisited-... Calculus Revisited: Multivariable Calculus | MIT https://ocw.mit.edu/resources/res-18-007-calculus-revisited-multivariable-calculus-fall-2011/index.htm https://ocw.mit.edu/resources/res-18-007-calculus-revisited-... Complex Variables, Differential Equations, and Linear Algebra | MIT https://ocw.mit.edu/resources/res-18-008-calculus-revisited-complex-variables-differential-equations-and-linear-algebra-fall-2011/index.htm https://ocw.mit.edu/resources/res-18-008-calculus-revisited-... Linear Algebra | MIT - https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD91010BC51F8 https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101... Introduction to Linear Dynamical Systems |Stanford https://see.stanford.edu/Course/EE263 https://see.stanford.edu/Course/EE263 Probability | Harvard https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6ht66KWxbzTIo https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6... Intermediate Statistics | CMU https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVWq-7wr8ffRtHF https://www.youtube.com/playlist?list=PLcW8xNfZoh7eI7KSWneVW... Convex Optimization I | Stanford https://see.stanford.edu/Course/EE364A https://see.stanford.edu/Course/EE364A Math Background for ML | CMU https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMWUaBRuLBvPBCg https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW... [0] https://news.ycombinator.com/item?id=14581962 https://news.ycombinator.com/item?id=14581962

ghosthamlet 9y ago

This book is aslo Mathematics for Machine Learning: https://github.com/soulmachine/machine-learning-cheat-sheet https://github.com/soulmachine/machine-learning-cheat-sheet

vigdals 9y ago

TheNewBoston's youtube channel also has some nice math videos that might help someone. Not sure if its applicable here. https://www.youtube.com/watch?v=UxhMTi2bh7k&list=PLAF739DF5F2D9506C https://www.youtube.com/watch?v=UxhMTi2bh7k&list=PLAF739DF5F...