8 ms·

Introduction to Linear Algebra for Applied Machine Learning with Python

425 points by Anon84 6y ago

hakcermani 6y ago
Thanks for this Pablo. I find this a good intro to LA as well as a tutorial on Numpy! Just one feedback would be good to have the TOC handy as we click around or a way to jump back to the top.
gandreani 6y ago
This is sick! I'm saving this for sure. I also really appreciate linking to other learning materials in the beginning (both free and non-free)
tomcat27 6y ago
are there resources where linear algebra is discussed in the context of deep learning?
- blackbear_ 6y ago
  "Linear Algebra and Learning from Data" by Gilbert Strang [1] has a section on that. https://math.mit.edu/~gs/learningfromdata/ https://math.mit.edu/~gs/learningfromdata/
- joshvm 6y ago
  Accompanying the book in the sibling comment is Strang's MIT 18.065 course which is exactly what you're after (it specifically covers linear algebra with a focus on data science and ML): https://ocw.mit.edu/courses/mathematics/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/ https://ocw.mit.edu/courses/mathematics/18-065-matrix-method... OCW so it's free(!), including the videos and handouts.
  - gspr 6y ago
    And Strang (more than) knows what he's talking about. Unlike OP, who treats minor corrections as personal attacks and goes all defensive. This is part of the reasons why it's dangerous to replace education from established institutions with random blogs on the internet.
    - tchaffee 6y ago
      > This is part of the reasons why it's dangerous to replace education from established institutions with random blogs on the internet. Please don't gatekeep education like this. I've caught teachers from established institutions in mistakes hundreds of times in my life, and often just as defensive. And I've found "random blogs" on the internet that explain things far clearer than the school recommended book. The only thing that is important is the quality of the resource, not where it came from. Strang's free videos are great. That's the important thing to share. (I edited to clarify which part of the parent comment I took issue with).
      - gspr 6y ago
        > Please don't gatekeep education like this. I've caught teachers from established institutions in mistakes hundreds of times in my life, and often just as defensive. And I've found "random blogs" on the internet that explain things far clearer than the school recommended book. Absolutely! And I've had medical doctors make lots of mistakes, while random blogs on the Internet have been right. Buuuut, you know. > The only thing that is important is the quality of the resource, not where it came from. Right. And the blog author we're discussing is flat out refusing to accept that his material has basic mistakes. That's a gigantic red flag. Again: it's not about being wrong. It's about refusing to accept that he's wrong.
        tchaffee 6y ago
        > Buuuut, you know. No I don't. Could you please explain? If random blogs have helped you catch doctors making mistakes (like they have me) then it would seem to strengthen my point that gatekeeping sources of valuable information based on their origin instead of their quality is a mistaken approach.
        gspr 6y ago
        What I mean is: even after experiencing that, I'm gonna see a medical doctor for medical problems. I bet you will too, if the problem is serious enough.
JPKab 6y ago
Great resource. I also want to post this series by Rachel Thomas of Fast.AI. https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6... She's a phenomenal teacher, and has grounding in the practice of applying it to ML and deep learning.
init-as 6y ago
Is there a website that anyone knows of for these types of open-source curriculums? Feel like it would be nice to have a place where people could post these and you could reference them when you wanted to learn something in the correct order and with the best resources.
- Guybrush_T 6y ago
  Something like Kahn Academy (https://www.khanacademy.org/ https://www.khanacademy.org/)?
  - init-as 6y ago
    Khan Academy is great, but I mean for all resources on a subject. In other words, a place where people can post curations of resources, not actual content. So say I wanted to learn linear algebra it would point to Khan Academy for the basics part and then other places as well.
    - uname_amiy 6y ago
      Something like this https://github.com/rossant/awesome-math https://github.com/rossant/awesome-math ?
- blackbear_ 6y ago
  The list of "awesome lists" on github [1] is generally quite helpful (haven't checked them all myself obviously). [1] https://github.com/sindresorhus/awesome https://github.com/sindresorhus/awesome
- mandliya 6y ago
  For AI/ML, I think https://madewithml.com/ https://madewithml.com/ does the job. It has curated open source projects, courses, papers and topics.
- PNWChris 6y ago
  They’re much narrower in scope than a full curriculum (or even a course, for that matter), but Better Explained [0] has some very good overviews of math topics. They are a useful supplement I’ve come across repeatedly while searching topics I found challenging over the years. On the topic of open source learning, I take every chance I can to heartily recommend fast.ai’s course [1]. It’s a good intro to Deep Learning that leaves you informed enough to build things, and equips you to ask follow-on questions and dive deeper when/where you need to. [0]: https://betterexplained.com/ https://betterexplained.com/ [1]: https://course.fast.ai/ https://course.fast.ai/
javier10e6 6y ago
Linear algebra: Everything is happy lad-dee-dah for x and y. The moment we go into matrices I go downhill from there. I saw a very cool youtube on Tesla Dojo and machine learning. I recommend.
- pbronez 6y ago
  One of the fun things about linear algebra is that you’re working with lots of simple equations all at once. It’s like moving from single variable calculus to multi-variable calculus. You realize that the additional complexity was there all the time, and you were just studying a special, restricted case before.
  - Jugurtha 6y ago
    Yes, and it's super useful. The first times you find yourself working with several equations and you 'just' put that into a matrix form and solve that system is an amazing feeling of 'truth' having been hiding in plain sight.
    - pbronez 6y ago
      Absolutely. It’s easy to forget how useful it is to be able to solve a system of equations like that. It lets you find optimal settings for complex systems with multiple constraints. Fitting a machine learning model is just one application for this... there’s a whole academic discipline called Operations Research that focuses on using optimization to find solutions to practical problems. Just another example of the huge power in creating accurate digital models of the world.
      - Jugurtha 6y ago
        We touched a bit of that. As an aside, when I was a student, we also had an applications module called "Numerical Analysis". We went through a bunch of algorithms (Cholesky, Seidel, Newton-Raphson, Gauss, etc.) and ran them by hand. We basically were "human computers". The exams were also solving problems using these algorithms, and given the iterative nature of many of them, if you make a tiny mistake it'll ripple and ruin everything for you. As an aside, my background is in optimal control/control theory and instrumentation, and operational calculus, state space representations, and matrices were a huge part of the "Jiu-Jitsu" we did. Tinkering with RST control, robust systems, finding optimal systems and sequences (Hamilton-Jacobi-Bellman-Pontryagin) with matrices flying around on our exam papers solving these systems. It was a nice abstraction. Very, very, powerful tools that command immense respect for Laplace, Lagrange, and people like them who invented things to solve problems we're facing now.
codebug88 6y ago
Hi, I did that article :) https://twitter.com/CodeBug88 https://twitter.com/CodeBug88 Saludos
- kobayashimaru 6y ago
  Ola
- tasuki 6y ago
  Is the 'sente' in your GitHub handle the opposite of 'gote', or something else entirely?
  - codebug88 6y ago
    Oh yes, I used to play a lot of Go/Baduk :)
gspr 6y ago
I was scrolling through to see if you explain how you typeset the math (notoriously hard on webpages), but I happened to instead find a small mistake. You write: "Elements of $\mathbb{R}^n$ are sets of real numbers." That is an incorrect definition. - It is inaccurate because n doesn't enter into the definition (and thus all Euclidean spaces would be the same). - You want ordered n-tuples, not sets. If you actually defined elements of R^n as sets of n real numbers, you would not be able to represent the diagonal (or, if you fix that, not be able to represent the difference between certain points on the diagonal of R^n and certain points on the diagonal of certain subspaces). Moreover, figure 5 talks about "overlapping vectors". This is highly non-standard, and definitely would need to be defined. Further, you're setting your readers up for trouble by defining vector arithmetic in terms of bases.
- codebug88 6y ago
  Keep in mind those are just notes I made for myself. I decided to put them out there just in case someone found them useful and to signal my skill... By practicing ML/DS in the last couple of years, I came to realize that what I put there is probably more than what is needed to know for applied ML/DS, and conceptual inaccuracies like those have little to none relevance
  - gspr 6y ago
    Well, wrong is wrong :-) I mean no offense by saying that, and it's human nature to be wrong. However, this is not a "conceptual inaccuracy". It's wrong. Straight up, old-fashioned wrong. And don't tell me it's of "little to no relevance" that your definition of R^2 does not distinguish between (0,1) and (1,0).
    - deleted 6y ago
      [deleted]
- gspr 6y ago
  Oh and perhaps more immediately: with this definition of R^n, (x,y) and (y,x) are the same point. No good.
BossingAround 6y ago
I just recently finally bought Strang's introduction to Linear Algebra. The book itself is very dense and, dare I say, scary to a newcomer, but his videos are amazingly clear in combination with the book. I am in the beginning, but so far, I'd recommend the book. Though, the steep price (almost 100USD over here) makes it a bit difficult to recommend. I might buy Boyd’s and Vandenberghe’s Introduction to Applied Linear Algebra later and write a comparison.
- nuclearnice1 6y ago
  As a heads up, the Boyd book is available free on his website. http://vmls-book.stanford.edu/ http://vmls-book.stanford.edu/
- egsmi 6y ago
  Jim Hefferon, who is also a HN member, has a great free Linear Algebra textbook. People all have their own preferences but I actually prefer his to Strang's. https://hefferon.net/linearalgebra/ https://hefferon.net/linearalgebra/
  - giu 6y ago
    +1 for Heffron's book, which was posted here 2 weeks ago [0]. He also published all the solutions to the exercises! [1] [0] https://news.ycombinator.com/item?id=24892907 https://news.ycombinator.com/item?id=24892907 [1] http://joshua.smcvt.edu/linearalgebra/jhanswer.pdf http://joshua.smcvt.edu/linearalgebra/jhanswer.pdf
- vector_spaces 6y ago
  My sense is that Vandenberghe is great for getting your feet wet and building intuition, but it's not as deep as Strang in its coverage and has fewer problems. Definitely still worth reading through especially since the problems lend themselves well to working through with Julia or Python or similar. For further reading, try Matrix Analysis and Applied Linear Algebra by Meyer.
Konohamaru 6y ago
It's not on the syllabus, but what application does jordan canonical form have to machine learning?
- Tainnor 6y ago
  probably none, since afaik the Jordan normal form is numerically very unstable. It's however quite useful in proofs (e.g. you can theoretically construct the matrix exponential out of it).
aanet 6y ago
This looks like an excellent resource. Thank you. The rendering of some of the Latex code does seem to be bit of a challenge; I see some intermingled code/text (on latest Mozilla on Ubuntu 18). The CPU usage also shoots up. What scripts are rendering the Latex code?
- codebug88 6y ago
  It usually takes a while to render the LaTeX code in my browser. If you scroll down immediately after entering the site you may find it looking weird. I just used mathjax syntax which supposedly renders on most browsers (my blog is hosted on GitHub, using Jekyll as site-generator). My knowledge of web-development is very basic, so I might be doing some things wrong ( ˘︹˘ )
dan-robertson 6y ago
I like that this touches on some of the algebra aspects. I always find it strange when I see something about “linear algebra” that never mentions linear maps, vector spaces or bases. Perhaps the mechanics of computing with matrices is better for an introduction for people who are only interested in machine learning but the mathematician in me feels like it is important to get across that in some specific sense the numbers in a matrix are arbitrary and that the essence of the thing is simpler. I’m a bit surprised that this doesn’t mention suffix notation, especially as multilinear maps are so common in machine learning and suffix notation is such a good way to work with them.
mrcactu5 6y ago
here's a Linear Algebra course from Columbia University with a bit of computer science emphasis. https://tonydear.github.io/teaching/coms3251 https://tonydear.github.io/teaching/coms3251
freakynit 6y ago
This is so wonderful. Thank you for putting this up :)

425 points by Anon84 6y ago

Thanks for this Pablo. I find this a good intro to LA as well as a tutorial on Numpy! Just one feedback would be good to have the TOC handy as we click around or a way to jump back to the top.

gandreani 6y ago

This is sick! I'm saving this for sure. I also really appreciate linking to other learning materials in the beginning (both free and non-free)

tomcat27 6y ago

are there resources where linear algebra is discussed in the context of deep learning?

blackbear_ 6y ago
"Linear Algebra and Learning from Data" by Gilbert Strang [1] has a section on that. https://math.mit.edu/~gs/learningfromdata/ https://math.mit.edu/~gs/learningfromdata/
joshvm 6y ago
Accompanying the book in the sibling comment is Strang's MIT 18.065 course which is exactly what you're after (it specifically covers linear algebra with a focus on data science and ML): https://ocw.mit.edu/courses/mathematics/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/ https://ocw.mit.edu/courses/mathematics/18-065-matrix-method... OCW so it's free(!), including the videos and handouts.
- gspr 6y ago
  And Strang (more than) knows what he's talking about. Unlike OP, who treats minor corrections as personal attacks and goes all defensive. This is part of the reasons why it's dangerous to replace education from established institutions with random blogs on the internet.
  - tchaffee 6y ago
    > This is part of the reasons why it's dangerous to replace education from established institutions with random blogs on the internet. Please don't gatekeep education like this. I've caught teachers from established institutions in mistakes hundreds of times in my life, and often just as defensive. And I've found "random blogs" on the internet that explain things far clearer than the school recommended book. The only thing that is important is the quality of the resource, not where it came from. Strang's free videos are great. That's the important thing to share. (I edited to clarify which part of the parent comment I took issue with).
    - gspr 6y ago
      > Please don't gatekeep education like this. I've caught teachers from established institutions in mistakes hundreds of times in my life, and often just as defensive. And I've found "random blogs" on the internet that explain things far clearer than the school recommended book. Absolutely! And I've had medical doctors make lots of mistakes, while random blogs on the Internet have been right. Buuuut, you know. > The only thing that is important is the quality of the resource, not where it came from. Right. And the blog author we're discussing is flat out refusing to accept that his material has basic mistakes. That's a gigantic red flag. Again: it's not about being wrong. It's about refusing to accept that he's wrong.
      - tchaffee 6y ago
        > Buuuut, you know. No I don't. Could you please explain? If random blogs have helped you catch doctors making mistakes (like they have me) then it would seem to strengthen my point that gatekeeping sources of valuable information based on their origin instead of their quality is a mistaken approach.
        gspr 6y ago
        What I mean is: even after experiencing that, I'm gonna see a medical doctor for medical problems. I bet you will too, if the problem is serious enough.

JPKab 6y ago

Great resource. I also want to post this series by Rachel Thomas of Fast.AI. https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6... She's a phenomenal teacher, and has grounding in the practice of applying it to ML and deep learning.

init-as 6y ago

Is there a website that anyone knows of for these types of open-source curriculums? Feel like it would be nice to have a place where people could post these and you could reference them when you wanted to learn something in the correct order and with the best resources.

Guybrush_T 6y ago
Something like Kahn Academy (https://www.khanacademy.org/ https://www.khanacademy.org/)?
- init-as 6y ago
  Khan Academy is great, but I mean for all resources on a subject. In other words, a place where people can post curations of resources, not actual content. So say I wanted to learn linear algebra it would point to Khan Academy for the basics part and then other places as well.
  - uname_amiy 6y ago
    Something like this https://github.com/rossant/awesome-math https://github.com/rossant/awesome-math ?
blackbear_ 6y ago
The list of "awesome lists" on github [1] is generally quite helpful (haven't checked them all myself obviously). [1] https://github.com/sindresorhus/awesome https://github.com/sindresorhus/awesome
mandliya 6y ago
For AI/ML, I think https://madewithml.com/ https://madewithml.com/ does the job. It has curated open source projects, courses, papers and topics.
PNWChris 6y ago
They’re much narrower in scope than a full curriculum (or even a course, for that matter), but Better Explained [0] has some very good overviews of math topics. They are a useful supplement I’ve come across repeatedly while searching topics I found challenging over the years. On the topic of open source learning, I take every chance I can to heartily recommend fast.ai’s course [1]. It’s a good intro to Deep Learning that leaves you informed enough to build things, and equips you to ask follow-on questions and dive deeper when/where you need to. [0]: https://betterexplained.com/ https://betterexplained.com/ [1]: https://course.fast.ai/ https://course.fast.ai/

javier10e6 6y ago

Linear algebra: Everything is happy lad-dee-dah for x and y. The moment we go into matrices I go downhill from there. I saw a very cool youtube on Tesla Dojo and machine learning. I recommend.

pbronez 6y ago
One of the fun things about linear algebra is that you’re working with lots of simple equations all at once. It’s like moving from single variable calculus to multi-variable calculus. You realize that the additional complexity was there all the time, and you were just studying a special, restricted case before.
- Jugurtha 6y ago
  Yes, and it's super useful. The first times you find yourself working with several equations and you 'just' put that into a matrix form and solve that system is an amazing feeling of 'truth' having been hiding in plain sight.
  - pbronez 6y ago
    Absolutely. It’s easy to forget how useful it is to be able to solve a system of equations like that. It lets you find optimal settings for complex systems with multiple constraints. Fitting a machine learning model is just one application for this... there’s a whole academic discipline called Operations Research that focuses on using optimization to find solutions to practical problems. Just another example of the huge power in creating accurate digital models of the world.
    - Jugurtha 6y ago
      We touched a bit of that. As an aside, when I was a student, we also had an applications module called "Numerical Analysis". We went through a bunch of algorithms (Cholesky, Seidel, Newton-Raphson, Gauss, etc.) and ran them by hand. We basically were "human computers". The exams were also solving problems using these algorithms, and given the iterative nature of many of them, if you make a tiny mistake it'll ripple and ruin everything for you. As an aside, my background is in optimal control/control theory and instrumentation, and operational calculus, state space representations, and matrices were a huge part of the "Jiu-Jitsu" we did. Tinkering with RST control, robust systems, finding optimal systems and sequences (Hamilton-Jacobi-Bellman-Pontryagin) with matrices flying around on our exam papers solving these systems. It was a nice abstraction. Very, very, powerful tools that command immense respect for Laplace, Lagrange, and people like them who invented things to solve problems we're facing now.

codebug88 6y ago

Hi, I did that article :) https://twitter.com/CodeBug88 https://twitter.com/CodeBug88 Saludos

kobayashimaru 6y ago
Ola
tasuki 6y ago
Is the 'sente' in your GitHub handle the opposite of 'gote', or something else entirely?
- codebug88 6y ago
  Oh yes, I used to play a lot of Go/Baduk :)

gspr 6y ago

I was scrolling through to see if you explain how you typeset the math (notoriously hard on webpages), but I happened to instead find a small mistake. You write: "Elements of $\mathbb{R}^n$ are sets of real numbers." That is an incorrect definition. - It is inaccurate because n doesn't enter into the definition (and thus all Euclidean spaces would be the same). - You want ordered n-tuples, not sets. If you actually defined elements of R^n as sets of n real numbers, you would not be able to represent the diagonal (or, if you fix that, not be able to represent the difference between certain points on the diagonal of R^n and certain points on the diagonal of certain subspaces). Moreover, figure 5 talks about "overlapping vectors". This is highly non-standard, and definitely would need to be defined. Further, you're setting your readers up for trouble by defining vector arithmetic in terms of bases.

codebug88 6y ago
Keep in mind those are just notes I made for myself. I decided to put them out there just in case someone found them useful and to signal my skill... By practicing ML/DS in the last couple of years, I came to realize that what I put there is probably more than what is needed to know for applied ML/DS, and conceptual inaccuracies like those have little to none relevance
- gspr 6y ago
  Well, wrong is wrong :-) I mean no offense by saying that, and it's human nature to be wrong. However, this is not a "conceptual inaccuracy". It's wrong. Straight up, old-fashioned wrong. And don't tell me it's of "little to no relevance" that your definition of R^2 does not distinguish between (0,1) and (1,0).
  - deleted 6y ago
    [deleted]
gspr 6y ago
Oh and perhaps more immediately: with this definition of R^n, (x,y) and (y,x) are the same point. No good.

BossingAround 6y ago

I just recently finally bought Strang's introduction to Linear Algebra. The book itself is very dense and, dare I say, scary to a newcomer, but his videos are amazingly clear in combination with the book. I am in the beginning, but so far, I'd recommend the book. Though, the steep price (almost 100USD over here) makes it a bit difficult to recommend. I might buy Boyd’s and Vandenberghe’s Introduction to Applied Linear Algebra later and write a comparison.

nuclearnice1 6y ago
As a heads up, the Boyd book is available free on his website. http://vmls-book.stanford.edu/ http://vmls-book.stanford.edu/
egsmi 6y ago
Jim Hefferon, who is also a HN member, has a great free Linear Algebra textbook. People all have their own preferences but I actually prefer his to Strang's. https://hefferon.net/linearalgebra/ https://hefferon.net/linearalgebra/
- giu 6y ago
  +1 for Heffron's book, which was posted here 2 weeks ago [0]. He also published all the solutions to the exercises! [1] [0] https://news.ycombinator.com/item?id=24892907 https://news.ycombinator.com/item?id=24892907 [1] http://joshua.smcvt.edu/linearalgebra/jhanswer.pdf http://joshua.smcvt.edu/linearalgebra/jhanswer.pdf
vector_spaces 6y ago
My sense is that Vandenberghe is great for getting your feet wet and building intuition, but it's not as deep as Strang in its coverage and has fewer problems. Definitely still worth reading through especially since the problems lend themselves well to working through with Julia or Python or similar. For further reading, try Matrix Analysis and Applied Linear Algebra by Meyer.

Konohamaru 6y ago

It's not on the syllabus, but what application does jordan canonical form have to machine learning?

Tainnor 6y ago
probably none, since afaik the Jordan normal form is numerically very unstable. It's however quite useful in proofs (e.g. you can theoretically construct the matrix exponential out of it).

aanet 6y ago

This looks like an excellent resource. Thank you. The rendering of some of the Latex code does seem to be bit of a challenge; I see some intermingled code/text (on latest Mozilla on Ubuntu 18). The CPU usage also shoots up. What scripts are rendering the Latex code?

codebug88 6y ago
It usually takes a while to render the LaTeX code in my browser. If you scroll down immediately after entering the site you may find it looking weird. I just used mathjax syntax which supposedly renders on most browsers (my blog is hosted on GitHub, using Jekyll as site-generator). My knowledge of web-development is very basic, so I might be doing some things wrong ( ˘︹˘ )

dan-robertson 6y ago

I like that this touches on some of the algebra aspects. I always find it strange when I see something about “linear algebra” that never mentions linear maps, vector spaces or bases. Perhaps the mechanics of computing with matrices is better for an introduction for people who are only interested in machine learning but the mathematician in me feels like it is important to get across that in some specific sense the numbers in a matrix are arbitrary and that the essence of the thing is simpler. I’m a bit surprised that this doesn’t mention suffix notation, especially as multilinear maps are so common in machine learning and suffix notation is such a good way to work with them.

mrcactu5 6y ago

here's a Linear Algebra course from Columbia University with a bit of computer science emphasis. https://tonydear.github.io/teaching/coms3251 https://tonydear.github.io/teaching/coms3251

freakynit 6y ago

This is so wonderful. Thank you for putting this up :)