4 ms·

Any suggestions on what to learn in Linear Algebra after Gilbert Strang’s 18.06SC? https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011/ https://ocw.mi

Any suggestions on what to learn in Linear Algebra after Gilbert Strang’s 18.06SC? https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011/ https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011... My goal is to learn the math behind machine learning.

deleted 4y ago
[deleted]
isaacfrond 4y ago
Here are some links from my favourite reference website: Book: Mathematics for Machine Learning (mml-book.github.io) https://news.ycombinator.com/item?id=16750789 https://news.ycombinator.com/item?id=16750789 Mathematics for Machine Learning [pdf] (mml-book.com) https://news.ycombinator.com/item?id=21293132 https://news.ycombinator.com/item?id=21293132 Mathematics of Machine Learning (2016) (ibm.com) https://news.ycombinator.com/item?id=15146746 https://news.ycombinator.com/item?id=15146746
natly 4y ago
These lectures are fantastic after you've mastered the basics of linear algebra https://www.youtube.com/watch?v=McLq1hEq3UY https://www.youtube.com/watch?v=McLq1hEq3UY (convex optimization, by a very experienced and often funny lecturer)
- nightski 4y ago
  Stephen Boyd is a very good lecturer! I watched his videos on linear dynamical systems almost a decade ago and thought he did a fantastic job. Would highly recommend.
  - sjburt 4y ago
    The problem set for the stanford linear dynamic systems that he taught is also fantastic. The level of difficulty and the breadth of applications leaves you with so many tools.
swframe2 4y ago
The math used in ML papers is very diverse. There is too much to learn. It is easier if you pick a problem and learn the math for it. Find a group that is already working on that problem and ask them for best way to learn the math for it.
- DavidSJ 4y ago
  A strong foundation in linear algebra, multivariable calculus, probability, and statistics is going to be generally applicable almost no matter what problem you work on.
ngmc 4y ago
You could take Strang’s follow-up course on learning from data. https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/ https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-an...
- jstx1 4y ago
  I don't know if 18.065 was worth the time, it felt like a repeat of 18.06 with hardly anything new (and not nearly enough context around actual applications).
ivad 4y ago
You might enjoy these notes: http://mlg.eng.cam.ac.uk/teaching/4f13/2122/ http://mlg.eng.cam.ac.uk/teaching/4f13/2122/ They give (I think) a good general overview, while also going a little bit more in-depth in a few areas (e.g., Gaussian Processes).
DavidSJ 4y ago
If you want a beautiful abstract perspective on linear algebra to complement Strang's more down-to-earth, matrix- and linear equation-oriented lectures, pick up Axler's Linear Algebra Done Right.
- natly 4y ago
  He also released a lecture series recently (Axler himself!) which barely anyone seems to be talking about: https://www.youtube.com/playlist?list=PLGAnmvB9m7zOBVCZBUUmSinFV0wEir2Vw https://www.youtube.com/playlist?list=PLGAnmvB9m7zOBVCZBUUmS...
screye 4y ago
I am a big fan of Murphy's ML book [1]. [1] https://smile.amazon.com/Probabilistic-Machine-Learning-Introduction-Computation/dp/0262046822?sa-no-redirect=1 https://smile.amazon.com/Probabilistic-Machine-Learning-Intr... It covers almost all the math you'd need to start doing ML research. I find it to be the ideal 'one book to rule them all' book for CS people in ML. Although, pure math grads in ML might find the book to not go deep enough.
time_to_smile 4y ago
The vast majority of the mathematics required to really understand ML is just probability, calculus and basic linear algebra. If you know these already and still struggle it's because the notation is often terse and assumes a specific context. Regarding this the only answer is to keep reading key papers and work through the math, ideally in code as well. For most current gen deep learning there's not even that much math, it's mostly a growing library of what are basically engineering tricks. For example an LSTM is almost exclusively very basic linear algebra with some calculus used to optimize it. But by it's nature the calculus can't be done by hand and the actual implementation of all that basic linear algebra is tricky and takes practice. You'll learn more by implementing things from scratch based on the math than you will trying to read through all the background material hoping that one day it will all make sense. It only ever makes sense when implementing and by continuous reading/practice.
- mbrudd 4y ago
  Amen!! This is the best way to learn anything technical -- put things into practice to understand the theory. It's also important to keep revisiting the theory to understand results, rather than parroting some catchphrase to "explain" results.

deleted 4y ago

[deleted]

isaacfrond 4y ago

Here are some links from my favourite reference website: Book: Mathematics for Machine Learning (mml-book.github.io) https://news.ycombinator.com/item?id=16750789 https://news.ycombinator.com/item?id=16750789 Mathematics for Machine Learning [pdf] (mml-book.com) https://news.ycombinator.com/item?id=21293132 https://news.ycombinator.com/item?id=21293132 Mathematics of Machine Learning (2016) (ibm.com) https://news.ycombinator.com/item?id=15146746 https://news.ycombinator.com/item?id=15146746

natly 4y ago

These lectures are fantastic after you've mastered the basics of linear algebra https://www.youtube.com/watch?v=McLq1hEq3UY https://www.youtube.com/watch?v=McLq1hEq3UY (convex optimization, by a very experienced and often funny lecturer)

nightski 4y ago
Stephen Boyd is a very good lecturer! I watched his videos on linear dynamical systems almost a decade ago and thought he did a fantastic job. Would highly recommend.
- sjburt 4y ago
  The problem set for the stanford linear dynamic systems that he taught is also fantastic. The level of difficulty and the breadth of applications leaves you with so many tools.

swframe2 4y ago

The math used in ML papers is very diverse. There is too much to learn. It is easier if you pick a problem and learn the math for it. Find a group that is already working on that problem and ask them for best way to learn the math for it.

DavidSJ 4y ago
A strong foundation in linear algebra, multivariable calculus, probability, and statistics is going to be generally applicable almost no matter what problem you work on.

ngmc 4y ago

You could take Strang’s follow-up course on learning from data. https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/ https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-an...

jstx1 4y ago
I don't know if 18.065 was worth the time, it felt like a repeat of 18.06 with hardly anything new (and not nearly enough context around actual applications).

ivad 4y ago

You might enjoy these notes: http://mlg.eng.cam.ac.uk/teaching/4f13/2122/ http://mlg.eng.cam.ac.uk/teaching/4f13/2122/ They give (I think) a good general overview, while also going a little bit more in-depth in a few areas (e.g., Gaussian Processes).

DavidSJ 4y ago

If you want a beautiful abstract perspective on linear algebra to complement Strang's more down-to-earth, matrix- and linear equation-oriented lectures, pick up Axler's Linear Algebra Done Right.

natly 4y ago
He also released a lecture series recently (Axler himself!) which barely anyone seems to be talking about: https://www.youtube.com/playlist?list=PLGAnmvB9m7zOBVCZBUUmSinFV0wEir2Vw https://www.youtube.com/playlist?list=PLGAnmvB9m7zOBVCZBUUmS...

screye 4y ago

I am a big fan of Murphy's ML book [1]. [1] https://smile.amazon.com/Probabilistic-Machine-Learning-Introduction-Computation/dp/0262046822?sa-no-redirect=1 https://smile.amazon.com/Probabilistic-Machine-Learning-Intr... It covers almost all the math you'd need to start doing ML research. I find it to be the ideal 'one book to rule them all' book for CS people in ML. Although, pure math grads in ML might find the book to not go deep enough.

time_to_smile 4y ago

The vast majority of the mathematics required to really understand ML is just probability, calculus and basic linear algebra. If you know these already and still struggle it's because the notation is often terse and assumes a specific context. Regarding this the only answer is to keep reading key papers and work through the math, ideally in code as well. For most current gen deep learning there's not even that much math, it's mostly a growing library of what are basically engineering tricks. For example an LSTM is almost exclusively very basic linear algebra with some calculus used to optimize it. But by it's nature the calculus can't be done by hand and the actual implementation of all that basic linear algebra is tricky and takes practice. You'll learn more by implementing things from scratch based on the math than you will trying to read through all the background material hoping that one day it will all make sense. It only ever makes sense when implementing and by continuous reading/practice.

mbrudd 4y ago
Amen!! This is the best way to learn anything technical -- put things into practice to understand the theory. It's also important to keep revisiting the theory to understand results, rather than parroting some catchphrase to "explain" results.