8 ms·
The matrix calculus you need for deep learning (2018)
- brailsafe 6y ago> We assume no math knowledge beyond what you learned in calculus 1 Anyone got recommendations for self-studying and testing Calculus 1?
- jonwachob91 6y ago1) go into knowing that the terminology is the hardest part to learn. The arithmetic is fairly easy, but the word salads can get daunting. 2) khan academy, coursera, MIT open courseware, whatever books your library has. Piece together multiple sources, b/c you don't have a professor to ask questions to, you'll need all the different sources and how each source explains the same idea differently for everything to 'click'.
- craigching 6y agoI refreshed on linear algebra and multi variable calculus through Khan Academy before I took a theory-based machine learning class. For me, it was a really good experience and I can’t overstate my appreciation for Sal Khan’s teaching style.
- 29athrowaway 6y agoTo begin, try to get a really high level explanation for differential and integral calculus. Look at videos, illustrations, etc. Then, once you have a mental image of what you will be doing, get into the math from the ground up.
- carlosf 6y agoWhen I was like 14 years old one one my math teachers explained to us what calculus was using zero equations. He made everything look really easy. At college, it was the complete opposite. My professor started with limits and convergence, didn't even bother to explain why we were studying the subject. Kudos for people who actually explain things.
- 29athrowaway 6y agoI never got a high level explanation for what calculus was because the business model of my school involved having a passing rate below 30%. If the passing rate was too high, teachers got fired no matter what the reason was.
- psalminen 6y agoMy high school physics teacher explained calculus to me by plotting a velocity curve, then saying the tangential line at a point (derivative) is acceleration and the area from 0 to that point under the curve (integral) was the distance travelled. Made all of calculus very easy for me to grok going forward.
- brailsafe 6y agoI missed the day where the word tangent must have been explained, then failed basic trigonometry and felt like an idiot for the rest of high school.
- ChefboyOG 6y agoI found outlier.org to be fantastic.
- mhh__ 6y agoI found the book "Calculus: a complete course" very nice. It's very cheap and extremely unpretentious while also giving you a subtle taste of a bunch of interesting applications (Deep learning... bleh, give me electromagnetism!)
- ABeeSea 6y agoThey are probably terrible for self-study, but I want to mention them anyway. I am a huge fan of David Bressoud’s Calculus (and analysis!) books. His first one doesn’t have exercises and his second one has a lot of physics. They are all heavy on written narrative and interlaced with history. I find all four of them absolutely fascinating. Calculus: https://www.amazon.com/Calculus-Reordered-History-Big-Ideas/dp/0691181314 https://www.amazon.com/Calculus-Reordered-History-Big-Ideas/... https://www.amazon.com/Second-Year-Calculus-Undergraduate-Mathematics-ebook/dp/B006Y30FUW https://www.amazon.com/Second-Year-Calculus-Undergraduate-Ma... Analysis: https://www.amazon.com/Approach-Analysis-Mathematical-Association-Textbooks/dp/0883857472 https://www.amazon.com/Approach-Analysis-Mathematical-Associ... https://www.amazon.com/Lebesgues-Integration-Mathematical-Association-Textbooks/dp/0521711835 https://www.amazon.com/Lebesgues-Integration-Mathematical-As...
- p_m_c 6y agoCalculus Made Easy - 1910. It is even entertaining. "What One Fool Can Do, Another Can. (Ancient Simian Proverb.)" https://calculusmadeeasy.org/ https://calculusmadeeasy.org/
- vishnugupta 6y agoBetter explained[1] articles that explain the “what” part, such as the meaning of “e”, from the first principles. I have a reasonable handle on Calculus-1 but those articles really helped me connect the dots. After reading them I realized that all I knew was “how” without having a clue about “what” [1] https://betterexplained.com/ https://betterexplained.com/
- beforeolives 6y agoNothing beats taking a good university class on it - https://ocw.mit.edu/courses/mathematics/18-01-single-variable-calculus-fall-2006/ https://ocw.mit.edu/courses/mathematics/18-01-single-variabl...
- deleted 6y ago[deleted]
- airocker 6y agois there an online course for maths? like undergrad maths?
- IdiocyInAction 6y agoMIT OCW has lectures for most undergrad CS/Engineering Math.
- cinntaile 6y agohttps://tutorial.math.lamar.edu/ https://tutorial.math.lamar.edu/ are pretty good, although they're mostly focused on the applied part.
- xzel 6y agoPaul's notes are great! Carried my butt through Calc 3 and PDEs, especially when the math for these classes get a little more formulaic (that is there is a pretty small set of reasonable questions you could possibly ask). His stuff is significantly more well done than any text book I had assigned for college math.
- ABeeSea 6y agoI’ve seen “The Infinite Napkin” passed around in math circles. I can’t vouch for it and I haven’t read much of it: https://web.evanchen.cc/napkin.html https://web.evanchen.cc/napkin.html
- wrycoder 6y agoEr, +1000 Man, if I had forty hours instead of forty minutes, I bet I could actually have explained this all. This book is my attempt at those forty hours. This project has evolved to more than just forty hours.
- yellowcake0 6y agoIf you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3) will be symmetric (the Hessian). then just do your best to apply the single variable rules of differentiation (product rule, chain rule, etc.), and then mess around a bit with the result until all the dimensions match up and such that your result matches the appropriate case (1)-(3). For any more complicated functions you encounter such as the determinant, you can just look up its derivative when needed.
- rademacher 6y agoI suggest this reference [1] for anyone needing to look something up. [1] https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf
- benrbray 6y agoI think it conceptually helps a lot to distinguish between row and column vectors, even if they are coalesce in any production code you might write. (0a) An (nxm) matrix A represents a linear transformation f(x)=Ax from R^m -> R^n (0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax (3) The derivative of a function f : R^m -> R^n is a linear transformation with the same "type" R^m -> R^n as the original function. This means it is an (nxm) matrix. The other two rules can be derived from these rules: (1) The derivative of a function f : R^m -> R is a (1xm) matrix, aka a row vector. (2) The derivative of a function f : R^n -> R is an (nx1) matrix, aka a column vector. This perspective is helpful because, aided by the chain rule, we can easily derive all the other matrix algebra rules on the fly. Distinguishing row / column vectors helps catch conceptual type errors in places where it's inappropriate to add vectors and covectors together.
- omedusyo 6y agoIn (2) I think you meant to write "f : R -> R^n"
- dang 6y agoIf curious, past threads: Matrix calculus for deep learning part 2 - https://news.ycombinator.com/item?id=23358761 https://news.ycombinator.com/item?id=23358761 - May 2020 (6 comments) Matrix Calculus for Deep Learning - https://news.ycombinator.com/item?id=21661545 https://news.ycombinator.com/item?id=21661545 - Nov 2019 (47 comments) The Matrix Calculus You Need for Deep Learning - https://news.ycombinator.com/item?id=17422770 https://news.ycombinator.com/item?id=17422770 - June 2018 (77 comments) Matrix Calculus for Deep Learning - https://news.ycombinator.com/item?id=16267178 https://news.ycombinator.com/item?id=16267178 - Jan 2018 (81 comments) Others?
- rdedev 6y agoThis helped me a lot in understanding the math behind gradient descent pretty well. The only problem I faced was when I tried to derive the equations for a vectorised implementation of a neural network. Certain derivatives, like the derivatives of matrix multiplication, are not obvious from the from the math presented in the paper. Had to dig through a bit to bridge that gap between non vectorised and vectorised implementations. By vectorised implementation of gradient descent, I mean one where gradient descent looks at more that one training samplenat a time when updating weights
- karpathy 6y agoYou do not need any matrix calculus for deep learning and micrograd (https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd), which implements backpropagation for neural nets in 100 lines of code, is a proof. Everything else is just vectorization.
- amelius 6y agoYes, you'll probably need more information-theory and statistics than linear algebra. The focus seems to be in the wrong place.
- pdevr 6y agoSide note: Terence Parr, one of the authors of this paper, also created AntLR.
- jules 6y agoDo people do it this way? Isn't using index notation + Einstein summation convention way easier and more powerful? You only need to remember two rules: 1. dx_i/dx_k = [i==j] where [i==j] is 0 if i != j and 1 if i == j 2. (AB)_ij = A_ik B_kj You don't even need the second rule if the function you want to differentiate is in index notation in the first place. The example of the other comment: f(x) = x^T A x = x_i A_ij x_j df/dx_k = d/dx_k (x_i A_ij x_j) = [i==k] A_ij x_j + x_i A_ij [j==k] = A_kj x_j + x_i A_ik. You can also skip the intermediate [i==k] step and immediately set the indices of other factors in a term equal. For example, differentiating with respect to A is immediate, even though the method from the article can't even handle it directly: df/A_kl = x_k x_l
- stevemadere 6y agoThe idea is great but the many notational typos are going to confuse the hell out of people who don't already know this math. (e.g. References to vector variables are often not bolded even as they are introducing bold as the notation for vectors, reusing x excessively as a variable in multiple contexts in the same expression, etc.) I went to graduate school for physics so I know all of this math cold but reading this paper caused me much confusion until I figured out they had a ton of typos in it. Caveat reader. Upon closer inspection, I realized they are not so much typos as poor UI implementation of the rendering. The visual difference between bold and italic when this page renders in either Chrome or Firefox is so subtle, it's very hard to see. Many of the equations and in-line variable references are rendered with svg so one cannot even inspect the page source to determine the intended font weight. Regardless, it's very confusing to read because of that. Not exactly what you want when trying to explain vector calculus. :) P.S. I'll try to submit some comments to the authors to see if they can get this cleared up before it causes too much confusion.