7 ms·
Matrix Calculus for Deep Learning
- cjhanks 9y agoBookmarked, this is a very nice reference I will need to read in greater depth.
- marrowgari 9y agoIf you're looking at this with the intention of getting started in Deep Learning and feeling overwhelmed by the math then Andrew Ng offers a great course on Coursera that goes over all of the formulas needed to calculate the forward propagation, loss computation, backward propagation, and gradient descent. Highly recommend it for anyone interested in breaking into the field of machine learning.
- ink404 9y agohttps://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning this one?
- marrowgari 9y agothis one... https://www.coursera.org/learn/neural-networks-deep-learning/home/welcome https://www.coursera.org/learn/neural-networks-deep-learning...
- ransom1538 9y agoIt is also all free on youtube: https://www.youtube.com/watch?v=UzxYlbK2c7E https://www.youtube.com/watch?v=UzxYlbK2c7E
- wuliwong 9y agoThey also have it all on Stanford's site with some other information and course materials. https://see.stanford.edu/Course/CS229 https://see.stanford.edu/Course/CS229
- marrowgari 9y agoThis is great. But I find the advantage of Coursera is that it incorporates quizzes and programming homework into the lectures reinforcing the learning. Also, the material is updated and more relevant to today's ML problems then his 2008 lectures on youtube.
- singularity2001 9y agoif you want to get started with machine learning you should just start with Keras and do some theory later.
- jph00 9y agoJeremy here. Here to answer any questions or comments that you have. But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish https://github.com/parrt/bookish . (We tried Katex, Mathjax, and pretty much everything else but nothing rendered everything properly). I've never found anything that introduces the necessary matrix calculus for deep learning clearly, correctly, and accessibly - so I'm happy that this now exists.
- jules 9y agoWhat do you think about the index notation physicists use for tensor calculus?
- jph00 9y agoI don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me. The only tensor notation I've been happy with is that used in J (http://www.jsoftware.com http://www.jsoftware.com), which is simple, flexible, and concise. There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS+BOOK/Kolda-Bader-SAND2007-6702.pdf http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...
- deleted 9y ago[deleted]
- jules 9y agoThe notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply: A_ij = B_ik C_kj Differentiating this with respect to variable l: ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj) By writing out indices you can just use the rules for scalar derivatives.
- peetle 9y agoSo great! Thanks Terence and Jeremy!
- bahram_banisadr 9y agoBeautifully straightforward - going to be referring to this for quick refreshers
- lprd 9y agoIn school, I didn't make it much past basic calculus/algebra. As a self-taught programmer (my highest level of education is a high-school diploma), I seriously wish I could go back and put more effort into math. I love looking at these types of topics, but I have absolutely no clue what I'm looking at. If anyone can recommend any books, courses, or any other material that starts from high-school level math, and gradually increases in complexity, I would love to look at it. Cheers :)
- j2kun 9y agoI am writing a book that I hope can serve such a purpose. Would you be interested in taking a look? If so, shoot me an email at mathintersectprogramming@gmail.com Fair warning, I have shown it to a programmer who claimed some level of "math phobia," and they said the first chapter was too difficult. I rewrote that chapter since, and I think it is better, but I could use some feedback :)
- jph00 9y agoWell... this paper is really designed to be accessible with just high school math, if you take your time (a few weeks or months) and follow the references. Any time it relies on some concept, it includes a link to learning more about that concept, and also has a link to a forum where you can ask questions if you get stuck. There's also a table of all notation used. If you give it a go and find you're not successful, I'd be interested to hear where is the first point where you got stuck and couldn't get unstuck, since that would suggest a need for us to improve our paper!
- lprd 9y agoHi Jeremy, I greatly appreciate your response! I will take a long look at this paper again and attempt to digest it. Thanks again!
- KSS42 9y agoI would start by watching the Linear Algebra and Calculus videos by 3Blue1Brown. This will give you an intuitive understanding of both. https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2x... https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53...
- sydl 9y agoThanks for this great contribution. I would like to be able to read the math in DL papers. (sorry I'm asking for something that it's too broad) 1) How much does this document cover the notations in those papers. 2) When I read a paper and if I am not sure what the math means, does that mean that I did not grok the subject yet, or the math presented in that paper goes beyond the math given in this Matrix Calculus document (assuming I studied well this document).
- pacbard 9y agoI would say try to get as much of the intuition behind the math as you can. Knowing what an equation means rather than fully understanding the Greek notation is what it matters for practice. If you want to substantially contribute to the theoretical CS literature, you will need to have a good handle of the notation (for obvious reasons). Note: I come from math and Econ, so the split between practitioner and theorists might be different for CS/ML.
- blt 9y agoWhile matrix derivatives are important, there is also a lot of other math in DL papers. In particular, a lot of the probability side concerns expectations, KL divergences, entropy, etc., which are all defined in terms of integrals or sums. You need undergraduate-level probability background.
- jph00 9y agoThe first 5 chapters of the Goodfellow deep learning book are a great resource for understanding the probability, linear algebra, optimization, and information theory you need to digest deep learning papers.
- zwieback 9y agoThanks so much for this. I have no interest in deep learning (at the moment) but I was working through some papers about the Lucas Kanade tracker and this paper explains some of the underlying math in just the right amount of detail. The authors usually show the beginning and end point and just say something like "using the chain rule" we arrive at ... It took me a while to understand what they were saying and this paper helps a lot. The math is super easy but keeping all the notation s and conventions in my head is hard, I've never seen it laid out this nicely before. Thanks!
- parrt 9y agoHiya. That's funny because it's exactly what caused us to write this article. Jeremy and I were working on an automatic differentiation tool and couldn't find any description of the appropriate matrix calculus that explained the steps. Everything is just providing the solution without the intervening steps. We decided to write it down so we never have to figure out the notation again. haha
- adyavanapalli 9y agoMatrix calculus is a bit screwy when you realize that there are two possible notations to represent matrix derivatives (numerator vs. denominator layout; numerator layout is used in this guide). Plus, the notation is not very "speaking" for doing calculations unless you commit to memory some basic results.. which is why, as a physicist, I would recommend working in tensor calculus notation during calculations, and translating back to matrix notation for writing the results.
- parrt 9y agoI was also surprised when I saw that there was no standard notation for Jacobian matrices. We use the numerator notation in the article, but point out that there are papers that use the denominator notation. I think I remember from engineering school that we used numerator notation so we stuck with that.
- improbable22 9y agoThis is what index notation is good for, and I encourage everyone to learn it. Jacobians are dx_a/dx_b, two indices, and clearly b belongs to the derivative. Whether it's rows or columns is an implementation detail of how you're storing these numbers. Index notation also seems natural for programming: an element A[i,j] or a slice Z[3,4,:] are precisely this.
- mewert 9y agoI agree. If your Matrix calculus involves more complicated use of derivative operators you can't treat it like linear algebra anymore. Better to break it down into something like tensor notation first and back to matrices at the end. https://en.wikipedia.org/wiki/Del https://en.wikipedia.org/wiki/Del. Specifically I was always confused by the material derivative of a vector field when presented as either Matrix or vector calculus. If you represent it in tensor notation ( or explicitly break it out as operations on basis vectors ) it works out nicely.
- kaffeinecoma 9y agoWow, this is really a great resource. I wish it had been available a few years back when I took the free online version of CS231n. The hardest part (for me, anyway) was the long-forgotten Calculus needed for backprop. Especially as applied to matrices. I struggled at the time to find accessible explanations of many of the matrix operations, and you seem to have it all laid out here. Thank you.
- calebh 9y agoFortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org http://www.matrixcalculus.org Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.
- parrt 9y agoWow! Great little calculator. Thanks for pointing us at it.
- BucketSort 9y agoIndeed, most tools surprisingly lack this ability. I was shocked when I needed to break my calculations down to a piece-wise form when doing matrix calc with SymPy.
- adyavanapalli 9y agoNote: This website presumes denominator layout, which is different from what is used in the guide.
- smrtinsert 9y agoThanks for this. Was taking Andrew Ng's course but the way he glosses over the calculus and then expects the student to understand the implications at end of lecture was a turn off so I dropped it. I hated the feeling I wasn't learning, just memorizing solutions.
- jph00 9y agoYou might prefer the approach at http://course.fast.ai http://course.fast.ai - all the concepts are taught with code, instead of math, and understanding is developed by running experiments.
- joshgel 9y agoI did both and found fast.ai so much easier to understand for someone without a background in math, like me.
- deleted 9y ago[deleted]
- vervez 9y agoI've taken 3/5 of his Deep Learning MOOC and thought the calculus, at least its implementation, was very well explained. Were you dissatisfied with the lack of depth or lack of explanation? Just curious.
- tw1010 9y agoLike, did we ever get this much math on a regular basis back a few years ago on HN? It's exciting to see how math is seeping into engineering culture.
- dbetteridge 9y agoFantastic content, thankyou. If I could make one request it would be a bit of margin/padding on the left of the body text. Would make it more readable on mobile.
- hinkley 9y agoSo are we turning machine learning into a Euclidean distance calculation [with] many dimensions, with different weights for each dimension? That’s... not that sexy. But at least it makes sense to anyone with an undergrad degree in CS or math, which is something neural networks never accomplished.
- gigatexal 9y agoReading this shows me just how little I know about advanced mathematics.
- letlambda 9y ago>For functions of a single parameter, /partial operator/ is equivalent to /full derivative operator/ (for sufficiently smooth functions). Are you pulling my leg here, or do I need to scrap my understanding of calculus? If they exist, how could they not be equivalent?
- poster123 9y agoThe book "Matrix Differential Calculus with Applications in Statistics and Econometrics, 3rd Edition" by Magnus and Neudecker, is available at http://www.janmagnus.nl/misc/mdc2007-3rdedition http://www.janmagnus.nl/misc/mdc2007-3rdedition (468 pages).